Artificial intelligence (ai) recognition of echocardiogram images to enhance a mobile ultrasound device

ABSTRACT

Artificial intelligence (AI) recognition of echocardiogram (echo) images by a mobile ultrasound device comprises receiving a plurality of the echo images captured by the ultrasound device, the ultrasound device including a display and a user interface (UI) that displays the echo images to a user, the echo images comprising 2D images and Doppler modality images of a heart. One or more neural networks process the echo images to automatically classify the echo images by view type. The view type of the echo images is simultaneously displayed in the UI of the ultrasound device along with the echo images. A report is generated showing the calculated measurements of features in the echo images. The report showing the calculated measurements is displayed on a display device.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation in-part of co-pending patentapplication Ser. No. 16/833,001, filed Mar. 27, 2020, which is acontinuation in-part of co-pending patent application Ser. No.16/216,929, filed Dec. 11, 2018, which issued on Apr. 28, 2020 as U.S.Pat. No. 10,631,828, both assigned to the assignee of the presentapplication and incorporated herein by reference.

BACKGROUND

Cardiovascular disease including heart failure is a major health problemaccounting for about 30% of human deaths worldwide. Heart failure isalso the leading cause of hospitalization in adults over the age of 65years. Echocardiography is an important diagnostic aid in cardiology forthe morphological and functional assessment of the heart. In a typicalpatient echocardiogram (echo) examination, a clinician called asonographer places an ultrasound device against the patient's chest tocapture a number of 2D images/videos of the patients' heart. Reflectedsound waves reveal the inner structure of the heart walls and thevelocities of blood flows. The ultrasound device position is variedduring an echo exam to capture different anatomical sections as 2Dslices of the heart from different viewpoints or views. The clinicianhas the option of adding to these 2D images a waveform captured fromvarious possible modalities including continuous wave Doppler, m-mode,pulsed wave Doppler and pulsed wave tissue Doppler. The 2D images/videosand Doppler modality images are typically saved in DICOM (DigitalImaging and Communications in Medicine) formatted files. Although thetype of modality is partially indicated in the metadata of the DICOMfile, the ultrasound device position in both the modality and 2D views,which is the final determinant of which cardiac structure has beenimaged, is not.

After the patient examination, a clinician/technician goes through theDICOM files, manually annotates heart chambers and structures like theleft ventricle (LV) and takes measurements of those structures. Theprocess is reliant on the clinicians' training to recognize the view ineach image and make the appropriate measurements. In a follow upexamination, a cardiologist reviews the DICOM images and measurements,compares them to memorized guideline values and make a diagnosis basedon the interpretation made from the echocardiogram.

The current workflow process for analyzing DICOM images, measuringcardiac structures in the images and determining, predicting andprognosticating heart disease is highly manual, time-consuming anderror-prone. Because the workflow process is so labor intensive, morethan 95% of the images available in a typical patient echocardiographicstudy are never annotated or quantified. The view angle or Dopplermodality type by which an image was captured is typically not labelled,which means the overwhelming majority of stored DICOMs from past patientstudies and clinical trials do not possess the basic structure andnecessary identification of labels to allow for machine learning on thisdata.

BRIEF SUMMARY

The disclosed embodiments provide methods and systems for artificialintelligence (AI) recognition of echocardiogram (echo) images by amobile ultrasound device. Aspects of exemplary embodiment include atleast one processor receiving a plurality of the echo images captured bythe ultrasound device, the ultrasound device including a display and auser interface (UI) that displays the echo images to a user, the echoimages comprising 2D images and Doppler modality images of a heart. Oneor more neural networks process the echo images to automaticallyclassify the echo images by view type. The view type of the echo imagesare simultaneously displayed in the UI of the ultrasound device alongwith the echo images. A report showing the calculated measurements offeatures in the echo images is generated using the neural networks. Thereport showing the calculated measurements is then displayed on adisplay device.

According to the method and system disclosed herein, the disclosedembodiments provide an automated clinical workflow that diagnoses heartdisease, while enhancing functionality of the mobile ultrasound device.

BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS

FIGS. 1A-1C are diagrams illustrating embodiments of a system forimplementing an automated clinical workflow diagnoses heart diseasebased on both cardiac biomarker measurements and AI recognition of 2Dand Doppler modality Echocardiographic images for automated measurementsand the diagnosis, prediction and prognosis of heart disease. FIG. 1Dillustrates a block diagram of an example point-of-care (POC) device formeasuring cardiac biomarkers.

FIG. 2 illustrates architectural layers of the echo workflow engine.

FIG. 3 is a flow diagram illustrating one embodiment of a process forperformed by the echo workflow engine to diagnose heart disease based onboth cardiac biomarker measurements and AI recognition of both 2D andDoppler modality echo images to perform automated measurements and thediagnosis, prediction and prognosis of heart disease.

FIG. 4A is a flow diagram illustrating details of the process forautomatically recognizing and analyze both 2D and Doppler modality echoimages to perform automated measurements and the diagnosis, predictionand prognosis of heart disease according to one embodiment.

FIG. 4B is a flow diagram illustrating advanced functions of the echoworkflow engine.

FIG. 5A is diagram illustrating an example 2D echo image.

FIG. 5B is diagram illustrating an example Doppler modality image.

FIGS. 6A-6K are diagrams illustrating some example view typesautomatically classified by the echo workflow engine.

FIG. 7 is a diagram illustrating an example 2D image segmented toproduce annotations indicating cardiac chambers, and an example Dopplermodality image segmented to produce a mask and trace waveform

FIGS. 8A and 8B are diagrams illustrating examples of findingsystolic/diastolic end points.

FIGS. 9A and 9B are diagrams illustrating processing of an imagingwindow in a 2D echo image and the automated detection of out of sectorannotations.

FIGS. 10A and 10B are diagrams graphically illustrating structuralmeasurements automatically generated from annotations of cardiacchambers in a 2D image, and velocity measurements automaticallygenerated from annotations of waveforms in a Doppler modality.

FIG. 11 is a diagram graphically illustrating measurements of globallongitudinal strain that were automatically generated from theannotations of cardiac chambers in 2D images.

FIG. 12A is a diagram graphically illustrating an example set of bestmeasurement data based on largest volume cardiac chambers and the savingof the best measurement data to a repository.

FIG. 12B is a diagram graphically illustrating the input ofautomatically derived measurements from a patient with normal LV EFmeasurements into a set of rules to determine a conclusion that thepatient has normal diastolic function, diastolic dysfunction, orindeterminate.

FIG. 13 is a diagram graphically illustrating the output ofclassification, annotation and measurement data to an example JSON file.

FIG. 14 is a diagram illustrating a portion of an example report showinghighlighting values that are outside the range of Internationalguidelines.

FIG. 15 is a diagram illustrating a portion of an example report of MainFindings that may be printed and/or displayed by the user.

FIG. 16A is a diagram showing a graph A plotting PCWP and HFpEF scores.

FIG. 16B is a diagram showing a graph B plotting CV mortality orhospitalization for HF.

FIG. 17 is a diagram illustrating a federated training platformassociated with the automated clinical workflow system.

FIG. 18 is a diagram illustrating an automated clinical workflow systemembodiment where the echo workflow engine is configured to provide amobile ultrasound device with automatic recognition and measurements of2D and Doppler modality Echocardiographic images.

FIG. 19 is a flow diagram for enhancing an ultrasound device to performAI recognition of echocardiogram images.

FIG. 20A is a diagram illustrating acquisition of echo images by themobile ultrasound device.

FIG. 20B is a diagram of a user interface displayed by the mobileultrasound device.

FIGS. 20C, 20D and 20E are diagrams illustrating example pages of areport showing calculated measurements of features in the echo images.

FIG. 21 is a flow diagram showing processing by the echo workflow enginein a connected configuration comprising the echo workflow client innetwork communication with the echo workflow engine executing on one ormore servers.

FIG. 22A is flow diagram illustrating an AI-based guidance process foran ultrasound device to improve capture of echo image views.

FIG. 22B is a diagram of a user interface displayed by the mobileultrasound device.

FIGS. 22C-22E are diagrams showing a progression of feedback indicationscontinuing from FIG. 22B.

DETAILED DESCRIPTION

The disclosed embodiments relate to artificial intelligence (AI)recognition of echocardiogram images to enhance a mobile ultrasounddevice. The following description is presented to enable one of ordinaryskill in the art to make and use the invention and is provided in thecontext of a patent application and its requirements. Variousmodifications to the exemplary embodiments and the generic principlesand features described herein will be readily apparent. The exemplaryembodiments are mainly described in terms of particular methods andsystems provided in particular implementations. However, the methods andsystems will operate effectively in other implementations. Phrases suchas “exemplary embodiment”, “one embodiment” and “another embodiment” mayrefer to the same or different embodiments. The embodiments will bedescribed with respect to systems and/or devices having certaincomponents. However, the systems and/or devices may include more or lesscomponents than those shown, and variations in the arrangement and typeof the components may be made without departing from the scope of theinvention. The exemplary embodiments will also be described in thecontext of particular methods having certain steps. However, the methodand system operate effectively for other methods having different and/oradditional steps and steps in different orders that are not inconsistentwith the exemplary embodiments. Thus, the present invention is notintended to be limited to the embodiments shown, but is to be accordedthe widest scope consistent with the principles and features describedherein.

The disclosed embodiments provide method and system for implementing asoftware-based automatic clinical workflow that diagnoses heart diseasebased on both cardiac biomarker measurements and AI recognition of 2Dand Doppler modality echocardiographic images. In embodiments, theclinical workflow performs diagnosis, prediction and prognosis of heartdisease, and can be deployed in workstation or mobile-basedpoint-of-care systems.

FIGS. 1A-1C are diagrams illustrating embodiments of a system forimplementing an automated clinical workflow diagnoses heart diseasebased on both cardiac biomarker measurements and AI recognition of 2Dand Doppler modality Echocardiographic images. FIG. 1A shows a basicstandalone configuration for the automated clinical workflow system 10Aand a connected configuration 10B. The automated clinical workflow 10Ais primarily implemented as a software application, referred to as echoworkflow engine 12, that executes on a computer 14 operating in astandalone setting, disconnected from other devices on network 26. Thecomputer 14 may be implemented in any form factor including aworkstation, desktop, notebook, laptop server or tablet capable ofrunning an operating system, such as Microsoft Windows® (e.g., Windows7®, Windows 10®), Apple macOS®, Linux®, Apple iOS®, Android®, and thelike.

The computer 14 may include typical hardware components (not shown)including a processor, input devices (e.g., keyboard, pointing device,microphone for voice commands, buttons, touchscreen, etc.), outputdevices (e.g., a display device, speakers, and the like), and wired orwireless network communication interfaces (not shown) for communication.The computer 14 may include internal computer-readable media, such asmemory (not shown) containing computer instructions comprising the echoworkflow engine 12, which implements the functionality disclosed hereinwhen executed by one or more computer processors.

The computer 14 may further include local internal storage for storingone or more databases 16 and an image file archive 18. In oneembodiment, the contents of the image file archive 18 includeechocardiogram image files (also referred to herein as echo images),which in some embodiments may be stored in DICOM (Digital Imaging andCommunications in Medicine) format.

In one embodiment, the computer 14 is in communication with peripheraldevices such a point-of-care (POC) device 25, an ultrasound imagingdevice 24, or both. The POC device 25 is capable of measuring cardiacbiomarkers in POC environments such as an emergency room, intensive careunit, physician's office, an ambulance, a patient setting, and remoteemergency sites, as explained below with reference with FIG. 1D.

The ultrasound imaging device 24 captures echocardiogram images of apatient's organ (e.g., a heart), which may then be stored as a patientstudy using the database 16 and image file archive 18. For example, thecomputer 14 may be located in a hospital or clinical lab environmentwhere Echocardiography is performed as a diagnostic aid in cardiologyfor the morphological and functional assessment of the heart. During atypical patient echocardiogram exam (referred to as a study), asonographer or technician places the ultrasound imaging device 24against the patient's chest to capture 2D echo images/videos of theheart to help diagnose the particular heart ailment. Measurements of thestructure and blood flows are typically made using 2D slices of theheart and the position of the ultrasound imaging device 24 is variedduring an echo exam to capture different anatomical sections of theheart from different viewpoints. The technician has the option of addingto these 2D echo images a waveform captured from various possiblemodalities including: continuous wave Doppler, m-mode, pulsed waveDoppler and pulsed wave tissue Doppler. The 2D images and Dopplerwaveform images may be saved as DICOM files. Although the type ofmodality is sometimes indicated in the metadata of the DICOM file, the2D view is not.

The computer 14 may further include removable storage devices such as anoptical disk 20 and/or a flash memory 22 and the like for storage of theecho images. In some embodiments, the removable storage devices may beused as an import source of echo images and related data structures intothe internal image file archive 18, rather than or in addition to, theultrasound imaging device 24. The removable storage devices may also beused as an archive for echocardiogram data stored in the database 16and/or the image file archive 18.

FIG. 1A also shows an advanced optional embodiment, referred to asconnected configuration 10B, where the computer 14 may be connectedthrough the network 26 and a router 28 to other DICOM based devices,such as DICOM servers 30, network file share devices 32, echoworkstations 34, and/or cloud storage services 36 hosting DICOM files.It should be understood that the servers 30 are also computerscomprising a processor, input and output devices, wired or wirelessnetwork communication interfaces for communication, andcomputer-readable media, such as memory containing computer instructionscomprising components of the echo workflow engine 12. In the connectedconfiguration 10B, several possible interactions with the database 16and the image file archive 18 are possible, as described below.

One possible interaction is to use the cloud storage services 36 as aninternal archive. In case of very large archives consisting of largeamounts of DICOM files, the computer 14 may not have sufficient storageto host all files and the echo workflow engine 12 may be configured touse external network storage of the cloud storage services 36 for filestorage.

Another possible interaction is to use the cloud storage services 36 asan import source by i) selecting a DICOM file set or patient study,which includes the DICOM and Doppler waveforms images and patient dataand examination information, including cardiac biomarker measurements.The patient study may also be selected by a reserved DICOMDIR fileinstance, from which the patient, exams and image files are read.

Yet a further possible interaction is to use the DICOM servers 30, thenetwork file share devices 32, echo workstations 34, and/or DICOMclients (of FIG. 1C) acting as DICOM servers (workstations withmodalities CFind, CMove and CStore) in order to retrieve patients, examsand images by performing a CFind operation, followed by a CMoveoperation, to request the remote device to send the cardiac biomarkermeasurements and/or images resulting from the CFind operation.

Referring now to FIG. 1B, a handheld configuration 10C for the automatedclinical workflow system is shown. In this embodiment, the computer 14of FIG. 1A is implemented as a handheld device 14′, such as a tablet ora mobile phone, connected to a wired or wireless portable ultrasoundscanner probe 24′ that transmits echo images to the handheld device 14′.In one such embodiment, the echo workflow engine 12 may be implementedas an application executed by the handheld device 14′. In anotherembodiment, the handheld device 14′ may transmit the echo images toanother computer/server executing the echo workflow engine 12 forprocessing.

FIG. 1C illustrates a software as a service (SaaS) configuration 10D forthe automated clinical workflow. In this embodiment, the echo workflowengine 12 is run on a server 40 that is in communication over thenetwork 26 with a plurality of client devices 42. In this embodiment,the server 40 and the echo workflow engine 12 may be part of athird-party service that provides automated measurements and thediagnosis, prediction and prognosis of heart disease to client devices(e.g., hospital, clinic, or doctor computers) over the Internet. Itshould be understood that although the server 40 is shown as a singlecomputer, it should be understood that the functions of server 40 may bedistributed over more than one server. In an alternative embodiment, theserver 40 and the echo workflow engine 12 of FIG. 1C may be implementedas a virtual entity whose functions are distributed over multiple clientdevices 42. Likewise, it should be understood that although the echoworkflow engine 12 is shown as a single component in each embodiment,the functionality of the echo workflow engine 12 may be separated into agreater number of modules/components.

Conventionally, after a patient examination where echo images arecaptured stored, a clinician/technician goes through the DICOM files,manually annotates heart chambers and structures and takes measurements,which are presented in a report. In a follow up examination, a doctorwill review the DICOM images and measurements, compare them to memorizedguideline values and make a diagnosis. Such a process is reliant on theclinicians' training to recognize the view and make the appropriatemeasurements so that a proper diagnosis can be made. Such a process iserror-prone and time consuming.

According to the disclosed embodiments, the echo workflow engine 12mimics the standard clinical practice of diagnosing heart disease of apatient by combining cardiac biomarker measurements and processing DICOMfiles of the patient using a combination of machine learning, imageprocessing, and DICOM workflow techniques to derive clinicalmeasurements, diagnose specific diseases, and prognosticate patientoutcomes, as described below. While an automated solution to echo imageinterpretation using machine learning has been previously proposed, thesolution fails to take cardiac biomarker measurements into account andonly analyzes 2D echo images and not Doppler modality waveform images.The solution also mentions disease prediction, but only attempts tohandle two diseases (hypertrophic cardiomyopathy and cardiacamyloidosis) and the control only compares normal patients to diseasedpatients.

The echo workflow engine 12 of the disclosed embodiments, however,improves on the automated solution by optionally combining cardiacbiomarker measurements with machine learning that automaticallyrecognizes and analyzes not only 2D echo images but also Dopplermodality waveform images in order to diagnose heart disease. The echoworkflow engine 12 is also capable of comparing patients havingsimilar-looking heart diseases (rather than comparing normal patients todiseased patients), and automatically identifies additional diseases,including both heart failure with reduced ejection fraction (HFrEF) andheart failure with preserved ejection fraction (HFpEF). HFrEF is knownas heart failure due to left ventricular systolic dysfunction orsystolic heart failure and occurs when the ejection fraction is lessthan 40%. HFpEF is a form of congestive heart failure where in theamount of blood pumped from the heart's left ventricle with each beat(ejection fraction) is greater than 50%. Finally, unlike the proposedautomated solution, the echo workflow engine 12 automatically takes intoaccount cardiac biomarker measurements.

Cardiac biomarkers are substances that are released into the blood whenthe heart is damaged or stressed. Measurements of these biomarkers areused to help diagnose acute coronary syndrome (ACS) and cardiacischemia, conditions associated with insufficient blood flow to theheart. Tests for cardiac biomarkers can also be used to help determine aperson's risk of having these conditions. Increases in one or morecardiac biomarkers in the blood can identify people with ACS or cardiacischemia, allowing rapid and accurate diagnosis and appropriatetreatment of their condition.

Example types of cardiac biomarkers include B-type natriuretic peptide(BNP) and N-terminal pro-brain natriuretic peptide (NT-proBNP),High-sensitivity C-reactive Protein (hs-CRP), Cardiac Troponin, CreatineKinase (CK), Creatine kinase-MB (CK-MB), and Myoglobin. Cardiacbiomarker tests are typically available to a health practitioner 24hours a day, 7 days a week with a rapid turn-around-time. Some of thetests are performed at the point of care (POC), e.g., in the emergencydepartment or at a patient's bedside.

A key reason for under-diagnosis of HF is the non-specificity ofpresenting symptoms and signs, necessitating objective diagnostic tests.The measurement of plasma natriuretic peptide (NP) concentration isrecommended by international guidelines for the initial diagnosis of HF,particularly in situations where echocardiography is not readilyavailable such as non-hospital settings and primary care. For instance,an N-terminal pro-brain natriuretic peptide (NT-proBNP) concentrationbelow 125 pg/mL has high negative predictive value and is recommendedfor ruling-out HF in non-acute settings. However, several cardiovascularand non-cardiovascular causes of elevated NPs weaken their positivepredictive value in HF. This is especially the case in HFpEF, whereatrial fibrillation, advanced age, renal failure and obesity are commoncomorbidities and importantly impede the interpretation of NPmeasurements. In such cases, the demonstration of objective cardiacdysfunction by echocardiography is mandated for the diagnosis of HF.

Echocardiography is needed to distinguish among the types of HF (HFpEFor HF with reduced ejection fraction [HFrEF])—a distinction that cannotbe made by raised NP levels alone and is critical for the selection ofappropriate therapies. Traditional echocardiography is highly manual,time consuming, error-prone, limited to specialists, and involves longwaiting times (e.g. up to 9 months in some areas of NHS Scotland).However, the Artificial Intelligence (AI) approached described hereinallows fully automated, fast and reproducible echocardiographic imageanalysis; turning a manual process of 30 minutes, 250 clicks, with 21%variability, into an AI-automated process taking 2 minutes, 1 click,with 0% variability. Such AI-enabled echocardiographic interpretationtherefore not only increases efficiency and accuracy, but also opens thedoor to decision support for non-specialists.

According the disclosed embodiments, a combination of circulatingcardiac and echocardiographic biomarkers represents an ideal diagnosticpanel for HF. Such combined interpretation of multimodal data was notpossible in the past since blood-based and imaging-based labs largelyfunctioned independent of each other. In the current era of linkedelectronic health records and picture archiving and communication system(PACS) in many hospitals, the development of true “companiondiagnostics” with combined interpretation of both blood and imagingbiomarkers is possible. Moreover, advancements in medical AI enable deeplearning models to be developed for greater diagnostic/predictiveprecision than ever achieved before. Automation of these algorithms,built into decision support tools for clinical application, has thepotential to transform the diagnosis of HF.

FIG. 1D illustrates a block diagram of an example point-of-care (POC)device for measuring cardiac biomarkers. As stated above, the POC device42A is capable of measuring cardiac biomarkers in POC environments suchas an emergency room, intensive care unit, physician's office, anambulance, a patient setting, and remote emergency sites. Inembodiments, a patient blood sample may be delivered to the POC device42A either through a strip reader 50 that receives an insert strip (notshown) containing the sample, or through a sample reader 52 thatreceives the sample from a syringe or a patient's finger. The POC device42A analyzes the blood sample for one or more cardiac biomarkers anddisplays the results on a display 54 within minutes. The POC device 42Acan store the results as well as wirelessly transmit the results toother systems/devices in the POC system, such as the echo workflowengine 12, and/or the results can be saved in the patient record orstudy.

In one embodiment, the POC device 42A measures at least B-typenatriuretic peptide (BNP) and/or N-terminal pro-brain natriureticpeptide (NT-proBNP), as shown. In one embodiment, the POC device 42A maymeasure only NT-proBNP. In another embodiment, the POC device 42A mayalso measure other cardiac biomarkers including High-sensitivityC-reactive Protein (hs-CRP), Cardiac Troponin, Creatine Kinase (CK),Creatine kinase-MB (CK-MB), and Myoglobin. In one specific embodiment,an example of the POC device 42A is the commercially available COBAS 232POC System™ by ROCHE.

The availability of point-of-care (POC) testing for both NT-proBNP andechocardiography (e.g., using mobile echo probes connected to handheldsmart devices as in FIG. 1B) enables the use of AI-enabled tools withinthe primary care or community setting. Indeed, the current COVID-19pandemic has highlighted the urgent need for such point-of-carecommunity-based testing in Recovery Plans to respond to COVID-19.

FIG. 2 illustrates architectural layers of the echo workflow engine 12.In one embodiment, the echo workflow engine 12 may include a number ofsoftware components such as software servers or services that arepackaged together in one software application. For example, the echoworkflow engine 12 may include a machine learning layer 200, apresentation layer 202, and a database layer 204.

The machine learning layer 200 comprises several neural networks toprocess incoming echo images and corresponding metadata. The neuralnetworks used in the machine learning layer may comprise a mixture ofdifferent classes or model types. In one embodiment, machine learninglayer 200 utilizes a first neural network to classify 2D images by viewtype, and uses a second set of neural networks 200B to both extractfeatures from Doppler modality images and to use the extracted featuresto classify the Doppler modality images by region (the neural networksused to extract features may be different than the neural network usedto classify the images). The first neural network 200A and the secondset of neural networks 200B may be implemented using convolutionalneural network (CNN) and may be referred to as classification neuralnetworks or CNNs.

Additionally, a third set of neural networks 200C, including adversarialnetworks, are employed for each classified 2D view type in order tosegment the cardiac chambers in the 2D images and produce segmented 2Dimages. A fourth set of neural networks 200D are used for eachclassified Doppler modality region in order to segment the Dopplermodality images to generate waveform traces. In additional embodiments,the machine learning layer 200 may further include a set of one or moreprediction CNNs for disease prediction and optionally a set of one ormore prognosis CNNs for disease prognosis (not shown). The third andfourth sets of neural networks 200C and 200D may be implemented usingCNNs and may be referred to as segmentation neural networks or CNNs.

In machine learning, a CNN is a class of deep, feed-forward artificialneural network typically use for analyzing visual imagery. Each CNNcomprises an input and an output layer, as well as multiple hiddenlayers. In neural networks, each node or neuron receives an input fromsome number of locations in the previous layer. Each neuron computes anoutput value by applying some function to the input values coming fromthe previous layer. The function that is applied to the input values isspecified by a vector of weights and a bias (typically real numbers).Learning in a neural network progresses by making incrementaladjustments to the biases and weights. The vector of weights and thebias are called a filter and represents some feature of the input (e.g.,a particular shape).

The machine learning layer 200 operates in a training mode to train eachof the CNNs 200A-200D prior to the echo workflow engine 12 being placedin an analysis mode to automatically recognize and analyze echo imagesin patient studies. In one embodiment, the CNNs 200A-200D may be trainedto recognize and segment the various echo image views using thousands ofecho images from an online public or private echocardiogram DICOMdatabase.

The presentation layer 202 is used to format and present information toa user. In one embodiment, the presentation layer is written in HTML 5,Angular 4 and/or JavaScript. The presentation layer 202 may include aWindows Presentation Foundation (WPF) graphical subsystem 202A forimplementing a light weight browser-based user interface that displaysreports and allows a user (e.g., doctor/technician) to edit the reports.The presentation layer 202 may also include an image viewer 202B (e.g.,a DICOM viewer) for viewing echo images, and a python server 202C forrunning the CNN algorithms and generating a file of the results inJavaScript Object Notation (JSON) format, for example.

The database layer 204 in one embodiment comprises a SQL database 204Aand other external services that the system may use. The SQL database204A stores patient study information for individual patient studies,including cardiac biomarker measurements input to the system. In someembodiments, the database layer 204 may also include the image filearchive 18 of FIG. 1.

FIG. 3 is a flow diagram illustrating one embodiment of a process forperformed by the echo workflow engine 12 to diagnose heart disease basedon both cardiac biomarker measurements and AI recognition of both 2D andDoppler modality echo images to perform automated measurements. Theprocess occurs once the echo workflow engine 12 is trained and placed inanalysis mode.

The process may begin by the echo workflow engine 12 receiving from amemory one or more patient studies comprising i) one or more cardiacbiomarker measurements derived from a patient sample, and ii) aplurality of echocardiogram images taken by an ultrasound device of apatient organ, such as a heart (block 300). In embodiments, the cardiacbiomarker measurements may be obtained directly from a local or remotesource, including from a handheld point-of-care (POC) device, such asthe POC device 42A shown in FIG. 1D. In another embodiment, the cardiacbiomarker measurements may be obtained through traditional lab test. Thecardiac biomarker measurements may be stored in a patient study and/oran archive, such as in an electronic medical record system (EMR) recordand the like. In one embodiment, the patient study may include 70-90images and videos.

A first module of the echo workflow engine 12 may be used to operate asa filter to separate the plurality of echocardiogram images according to2D images and Doppler modality images based on analyzing image metadata(block 302). The first module analyzes the DICOM tags, or metadata,incorporated in the image, and runs an algorithm based upon the taginformation to distinguish between 2D and modality images, and thenseparate the modality images into either pulse wave, continuous wave,PWTDI or m-mode groupings. A second module of the echo workflow engine12 may perform color flow analysis on extracted pixel data using acombination of analyzing both DICOM tags/metadata and color contentwithin the images, to separate views that contain color from those thatdo not. A third module then anonymizes the data by removing metatagsthat contain personal information and cropping the images to exclude anyidentifying information. A fourth module then extracts the pixel datafrom the images and converts the pixel data to numpy arrays for furtherprocessing.

Because sonographers do not label the view types in the echo images, oneor more of neural networks are used classify the echo images by viewtype. In one embodiment, a first neural network is used by the echoworkflow engine 12 to classify the 2D images by view type (block 304);and a second set of neural networks is used by the echo workflow engine12 to extract the features from Doppler modality images and to use theextracted features to classify the Doppler modality images by region(block 306). As shown, the processing of 2D images is separate from theprocessing of Doppler modality images. In one embodiment, the firstneural network and the second set of neural networks may implementedusing the set of classification convolutional neural networks (CNNs)200A. In one specific embodiment, a five class CNN may be used toclassify the 2D images by view type and an 11 class CNN may be used toclassify the Doppler modality images by region. In one embodiment, aplurality of each of type of neural network can be implemented andconfigured to use a majority voting scheme to determine the optimalanswer. For example a video can be divided into still image frames, andeach frame may be given a classification label, i.e., of a vote, and theclassification label receiving the most votes is applied to classify thevideo.

In one embodiment, the echo workflow engine 12 is trained to classifymany different view types. For example, the echo workflow engine 12 mayclassify at least 11 different view types including parasternal longaxis (FLAX), apical 2-, 3-, and 4-chamber (A2C, A3C, and A4C), A4C pluspulse wave of the mitral valve, A4C plus pulse wave tissue Doppler onthe septal side, A4C plus pulse wave tissue Doppler on the lateral side,A4C plus pulse wave tissue Doppler on the tricuspid side, A5C pluscontinuous wave of the aortic valve, A4C+Mmode (TrV), A5C+PW (LVOT).

Based on the classified images, a third set of neural networks is usedby the echo workflow engine 12 to segment regions of interest (e.g.,cardiac chambers) in the 2D images to produce annotated or segmented 2Dimages (block 308). A fourth set of neural networks is used by the echoworkflow engine 12 for each classified Doppler modality region togenerate waveform traces and to generate annotated or segmented Dopplermodality images (block 309). The process of segmentation includesdetermining locations where each of the cardiac chambers begin and endto generate outlines of structures of the heart (e.g., cardiac chambers)depicted in each image and/or video. Segmentation can also be used totrace the outline of the waveform depicting the velocity of blood flowin a Doppler modality. In one embodiment, the third and fourth sets ofneural networks maybe referred to as segmentation neural networks and mycomprise the set of segmentation CNNs 200B and 200C. The choice ofsegmentation CNN used is determined by the view type of the image, whichmakes the prior correct classification of view type a crucial step. In afurther embodiment, once regions of interest are segmented, a separateneural network can be used to smooth outlines of the segmentations.

As will be explained further below, the segmentation CNNs may be trainedfrom hand-labeled real images or artificial images generated by generaladversarial networks (GANs).

Using both the segmented 2D images and the segmented Doppler modalityimages, the echo workflow engine 12 calculates for the patient study,measurements of cardiac features for both left and right sides of theheart (block 310).

The echo workflow engine 12 then generates conclusions by comparing theone or more cardiac biomarker measurements and calculated measurementsof cardiac features with International cardiac guidelines (block 312).The echo workflow engine 12 further outputs at least one report to auser showing ones of the one or more cardiac biomarker measurements andthe calculated measurements that fall within or outside of theInternational cardiac guidelines (block 314). In one embodiment, tworeports are generated and output: the first report is a list of thecardiac biomarker measurements and the calculated values for eachmeasurement with the highest confidence as determined by a rules basedengine, highlighting values among the measurements that fall outside ofthe International guidelines; and the second report is a comprehensivelist of all cardiac biomarker measurements and echo image measurementscalculated on every image frame of every video, in every view,generating large volumes of data. All report data and extracted pixeldata may be stored in a structured database to enable machine learningand predictive analytics on images that previously lacked thequantification and labelling necessary for such analysis. The structureddatabase may be exported to a cloud based server or may remain onpremises (e.g., of the lab owning the images) and can be connected toremotely. By connecting these data sources into a single network,disease prediction algorithms can be progressively trained acrossmultiple network nodes, and validated in distinct patient cohorts. Inone embodiment, the reports may be electronically displayed to a doctorand/or a patient on a display of an electronic device and/or as a paperreport. In some embodiments, the electronic reports may be editable bythe user per rule or role based permissions, e.g., a cardiologist may beallowed to modify the report, but a patient may have only viewprivileges.

FIG. 4A is a flow diagram illustrating further details of the processfor automatically recognizing and analyze both 2D and Doppler modalityecho images to perform automated measurements and the diagnosis,prediction and prognosis of heart disease according to one embodiment.

The process may begin with receiving one or more patient studies (FIG. 3block 300), which comprises blocks 400-4010. In one embodiment, echoimages from each of the patient studies are automatically downloadedinto the image file archive 18 (block 400). The cardiac biomarkermeasurements and the echo images may be received from a local or remotestorage source of the computer 14. The local storage sources may includeinternal/external storage of the computer 14 including removable storagedevices. The remote storage sources may include the ultrasound imagingdevice 24, the POS device 25, the DICOM servers 30, the network fileshare devices 32, the echo workstations 34, and/or the cloud storageservices 36 (see FIG. 1). In one embodiment, the echo workflow engine 12includes functions for operating as a picture archiving andcommunication server (PACS), which is capable of handling images frommultiple modalities (source machine types, one of which is theultrasound imaging device 24). The echo workflow engine 12 uses PACS todownload and store the echo images into the image file archive 18 andprovides the echo workflow engine 12 with access to the echo imagesduring the automated workflow. The format for PACS image storage andtransfer is DICOM (Digital Imaging and Communications in Medicine).

Patient information, including any cardiac biomarker measurements, fromeach of the patient studies is extracted and stored in the database 16(block 402). Non-image patient data may include metadata embedded withinthe DICOM images and/or scanned documents, which may be incorporatedusing consumer industry standard formats such as PDF (Portable DocumentFormat), once encapsulated in DICOM. In one embodiment, received patientstudies are placed in a processing queue for future processing, andduring the processing of each patient study, the echo workflow engine 12queues and checks for unprocessed echo images (block 404). The echoworkflow engine 12 monitors the status of patient studies, and keepstrack of them in a queue to determine which have been processed andwhich are still pending. In one embodiment, prioritization of thepatient studies in the queue may be configured by a user. For example,the patient studies may be prioritized in the queue for processingaccording to the date of the echo exam, the time of receipt of thepatient study or by estimated severity of the patient's heart disease.

Any unprocessed echo images are then filtered for having a valid DICOMimage format and non DICOM files in an echo study are discarded (block406). In one embodiment, the echo images are filtered for having aparticular type of format, for example, a valid DICOM file format, andany other file formats may be ignored. Filtering the echo images forhaving a valid image file format enhances the reliability of the echoworkflow engine 12 by rejecting invalid DICOM images for processing.

Any unprocessed valid echo images are then opened and processed in thememory of the computer 14 (block 408). Opening of the echo images forthe patient study in memory of the computer 14 is done to enhanceprocessing speed by echo workflow engine 12. This is in contrast to anapproach of opening the echo files as sub-processes, saving the echofiles to disk, and then reopening each echo image during processing,which could significantly slow processing speed.

The echo workflow engine 12 then extracts and stores the metadata fromthe echo images and then anonymizes the images by blacking out theimages and overwriting the metadata in order to protect patient dataprivacy by covering personal information written on the image (block410). As an example, DICOM formatted image files include metadatareferred to as DICOM tags that may be used to store a wide variety ofinformation such as patient information, Doctor information, ultrasoundmanufacture information, study information, and so on. In oneembodiment, the extracted metadata may be stored in the database 16 andthe metadata in image files is over written for privacy.

After receipt and processing of the patient studies, the echo workflowengine 12 separates 2D images from Doppler modality images so the twodifferent image types can be processed by different pipeline flows,described below. In one embodiment, the separating of the images (FIG. 3block 302) may comprise blocks 412-414. First, the 2D images areseparated from the Doppler modality images by analyzing the metadata(block 412).

FIGS. 5A and 5B are diagrams illustrating an example 2D echo image 500and an example Doppler modality image 502 including a waveform,respectively. The echo workflow engine 12 may determine the image typeby examining metadata/DICOM tags. In one embodiment, information withinthe DICOM tags may be extracted in order to group the images into one ofthe following four classes: 2D only, pulsed-wave (PW), continuous wave(CW), and m-mode. Similarly, the transducer frequency of the ultrasoundimaging device 24 in the metadata may be used to further separate someof the PW images into a fifth class: pulsed-wave tissue doppler imaging(PWTDI).

Referring again to FIG. 4A, the echo workflow engine 12 may also filterout images with a zoomed view, which may also be determined by analyzingthe metadata (block 414). Any of the echo images that has been zoomedduring image capture are not processed through the pipeline because whenzooming, useful information is necessarily left out of the image,meaning the original image would have to be referenced for the missingdata, which is a duplication of effort that slows processing speed.Accordingly, rather than potentially slowing the process in such amanner, the echo workflow engine 12 filters out or discards the zoomedimages to save processing time. In an alternative embodiment, filteringout zoomed images in block 414 may be performed prior to separating theimages in block 412.

After separating the 2D images from the Doppler modality images, theecho workflow engine 12 extracts and converts the image data from eachecho image into numerical arrays 416 (block 416). For the echo imagesthat are 2D only, the pixel data comprises a series of image framesplayed in sequence to create a video. Because the image frames areunlabeled, the view angle needs to be determined. For the Dopplermodality images that include waveform modalities, there are two imagesin the DICOM file that may be used for subsequent view identification, awaveform image and an echo image of the heart. The pixel data isextracted from the DICOM file and tags in the DICOM file determine thecoordinates to crop the images. The cropped pixel data is stored innumerical arrays for further processing. In one embodiment, blocks 412,414 and 416 may correspond to the separating images block 302 of FIG. 3.

After separating images, the echo workflow engine 12 attempts toclassify each of the echo images by view type. In one embodiment, viewclassification (FIG. 3 blocks 304 and 306) correspond to blocks 418-422.

According to the disclosed embodiments, the echo workflow engine 12attempts to classify each of the echo images by view type by utilizingparallel pipeline flows. The parallel pipeline includes a 2D imagepipeline and a Doppler modality image pipeline. The 2D pipeline flowbegins by classifying, by a first CNN, the 2D images by view type (block418), corresponding to block 304 from FIG. 3. The Doppler modality imagepipeline flow begins by classifying, by a second CNN, the Dopplermodality images by view type (block 420), corresponding to block 306from FIG. 3.

FIGS. 6A-6K are diagrams illustrating some example view typesautomatically classified by the echo workflow engine 12. As statedpreviously, example view types may include parasternal long axis (PLAX),apical 2-, 3-, and 4-chamber (A2C, A3C, and A4C), A4C plus pulse wave ofthe mitral valve, A4C plus pulse wave tissue Doppler on the septal side,A4C plus pulse wave tissue Doppler on the lateral side, A4C plus pulsewave tissue Doppler on the tricuspid side, A5C plus continuous wave ofthe aortic valve, A4C+Mmode (TrV), A5C+PW (LVOT).

Referring again to FIG. 4A, in one embodiment, 2D image classificationis performed as follows. If the DICOM file contains video frames from a2D view, only a small subset of the video frames are analyzed todetermine 2D view classification for more efficient processing. In oneembodiment, the subset of the video frames may range approximately 8-12video frames, but preferably 10 frames are input into one of the CNNs200A trained for 2D to determine the actual view. In an alternativeembodiment, subset a video frames may be randomly selected from thevideo file. In one embodiment, the CNNs 200A classify each of theanalyzed video frames as one of: A2C, A3C, A4C, A5C, PLAX Modified,PLAX, PSAX AoV level, PSAX Mid-level, Subcostal Ao, Subcostal Hep vein,Subcostal IVC, Subcostal LAX, Subcostal SAX, Suprasternal and Other.

Doppler modality images comprise two images, an echocardiogram image ofthe heart and a corresponding waveform, both of which are extracted fromthe echo file for image processing. In one embodiment, Doppler modalityimage classification of continuous wave (CW), pulsed-wave (PW), andM-mode images is performed as follows. If the DICOM file contains awaveform modality (CW, PW, PWTDI, M-mode), the two extracted images areinput to one of the CNNs 200A trained for CW, PW, PWTDI and M-mode viewclassification to further classify the echo images as one of: CW (AoV),CW (TrV), CW Other, PW (LVOT), PW (MV), PW Other, PWTDI (lateral), PWTDI(septal), PWTDI (tricuspid), M-mode (TrV) and M-mode Other.

There are many more potential classifications available for modalities,but the present embodiments strategically select the classes above,while grouping the remaining potential classes into “Other”, in order tomaximize processing efficiency, while identifying the most clinicallyimportant images for further processing and quantification.Customization of the CNNs 200A occurs in the desired number of layersused and the quantity of filters within each layer. During the trainingphase, the correct size of the CNNs may be determined through repeatedtraining and adjustments until optimal performance levels are reached.

During view classification, the echo workflow engine 12 maintainsclassification confidence scores that indicate a confidence level thatthe view classifications are correct. The echo workflow engine 12filters out the echo images having classification confidence scores thatfail to meet a threshold, i.e., low classification confidence scores(block 422). Multiple algorithms may be employed to deriveclassification confidence scores depending upon the view in question.Anomalies detected in cardiac structure annotations, image quality,cardiac cycles detected and the presence of image artifacts may allserve to decrease the classification confidence score and discard anecho image out of the automated echo workflow.

With respect to the confidence scores, the echo workflow engine 12generates and analyzes several different types of confidence scores atdifferent stages of processing, including classification, annotation,and measurements (e.g., blocks 422, 434 and 442). For example, poorquality annotations or classifications, which may be due to substandardimage quality, are filtered out by filtering the classificationconfidence scores. In another example, in a patient study the same viewmay be acquired more than once, in which case the best measurements arechosen by filtering out low measurement confidence scores as describedfurther below in block 442. Any data having a confidence score thatmeets a predetermined threshold continues through the workflow. Shouldthere be a duplication of measurements both with high confidence, themost clinically relevant measurement may be chosen.

Next the echo workflow engine 12 performs image segmentation to defineregions of interest (ROI). In computer vision, image segmentation is theprocess of partitioning a digital image into multiple segments (sets ofpixels) to locate and boundaries (lines, curves, and the like) ofobjects. Typically, annotations are a series of boundary linesoverlaying overlaid on the image to highlight segment boundaries/edges.In one embodiment, the segmentation to define ROI (FIG. 3 block 308)corresponds to blocks 426-436.

In one embodiment, the 2D image pipeline annotates, by a third CNN,regions of interests, such as cardiac chambers in the 2D images, toproduce annotated 2D images (block 426). An annotation post process thenerodes the annotations to reduce their dimensions, spline fits outlinesof cardiac structures and adjusts locations of the boundary lines closerto the region of interest (ROIs) (block 427). The 2D image pipelinecontinues with analyzing the ROIs (e.g., cardiac chambers) in theannotated 2D images to estimate volumes and determine key points in thecardiac cycle by finding systolic/diastolic end points (block 430). For2D only views, measurements are taken at the systolic or diastolic phaseof the cardiac cycle, i.e. when the left ventricle reaches the smallestvolume (systole) or the largest volume (diastole). From the 2D videoimages, it must be determined which end points are systolic and whichare diastolic based on the size of the estimated volumes of the leftventricle. For example a significantly large left ventricle may indicatea dystonic end point, while a significantly small volume may indicate asystolic end point. Every video frame is annotated and the volume of theleft ventricle is calculated throughout the whole cardiac cycle. Theframes with minimum and maximum volumes are detected with a peakdetection algorithm.

The Doppler modality pipeline analyzes the Doppler modality images andgenerates, by a fourth CNN, a mask and a waveform trace in the Dopplermodality images to produce annotated Doppler modality images (block431).

FIG. 7 is a diagram illustrating an example 2D image segmented in block426 to produce annotations 700 indicating cardiac chambers, and anexample Doppler modality image segmented in block 431 to produce a maskand waveform trace 702.

In one embodiment, the third and fourth CNNs may correspond tosegmentation CNNs 200B. In one embodiment, each of the CNNs 200B used tosegment the 2D images and Doppler modality images may be implemented asU-Net CNN, which is convolutional neural network developed forbiomedical image segmentation. Multiple U-Nets may be used. For example,for 2D images, a first U-Net CNN can be trained to annotate ventriclesand atria of the heart from the A2C, A3C, A4C views. A second U-net CNNcan be trained to annotate the chambers in the PLAX views. For M-modeviews, a third U-Net CNN can be trained to segment the waveform, removesmall pieces of the segments to find likely candidates for the region ofinterest, and then reconnect the segments to provide a full trace of themovement of the mitral valve. For CW views, a fourth U-net CNN can betrained to annotate and trace blood flow. For PW views, a fifth U-netCNN trained to annotate and trace the blood flow. For PWTDI views, asixth U-net CNN can be trained to annotate and trace movement of thetissues structures (lateral/septal/tricuspid valve).

Referring again to FIG. 4A, the Doppler modality pipeline continues byprocessing the annotated Doppler modality images with a sliding windowto identify cycles, peaks are measured in the cycles, and key points inthe cardiac cycle are determined by finding systolic/diastolic endpoints (block 432). Typically a Doppler modality video may capture threeheart cycles and the sliding window is adjusted in size to block out twoof the cycles so that only one selected cycle is analyzed. Within theselected cycle, the sliding window is used to identify cycles, peaks aremeasured in the cycles, and key points in the cardiac cycle aredetermined by finding systolic/diastolic end points.

FIGS. 8A and 8B are diagrams illustrating examples of findingsystolic/diastolic end points in the cardiac cycle, for both 2D andDoppler modalities, respectively, which are key points in order to takeaccurate cardiac measurements.

Referring again to FIG. 4A, in one embodiment, the echo workflow engine12 maintains annotation confidence scores corresponding to the estimatedvolumes, systolic/diastolic end points, identified cycles and measuredpeaks. The echo workflow engine 12 filters out annotated images havingannotation confidence scores that fail to meet a threshold, i.e., lowannotated confidence scores (block 434). Examples of low confidenceannotations may include annotated images having one or more of:excessively small areas/volumes, sudden width changes, out of proportionsectors, partial heart cycles, and insufficient heart cycles.

After images having low annotation confidence scores are filtered out,the echo workflow engine 12 defines an imaging window for each image,and filters out annotations that lie outside of the imaging window(block 435).

FIGS. 9A and 9B are diagrams illustrating processing of an imagingwindow in a 2D echo image 900. In FIG. 9A, a shaded ROI 902 is shownhaving unshaded regions or holes 904 therein. Open CV morphologytransformations are used to fill the holes 904 inside the ROI, as shownin In FIG. 9B. Thereafter, a Hough line transformation may be used tofind an imaging window border 906. As is well known, a Hough transformis a feature extraction technique used in digital image processing tofind imperfect instances of objects within a certain class of shapes.After an imaging window border is are found, a pixel account ofannotations beyond the imaging window border is made. Annotations 908with a significant number of pixels outside the border of the imagingwindow are then discarded.

Referring again to FIG. 4A, in the handheld configuration for theautomated clinical workflow system 10C (See FIG. 1B), the patientstudies may not include Doppler modality images. According to a furtheraspect, the disclosed embodiments accommodate for such handheldconfigurations by using the 2D images to simulate Doppler modalitymeasurements by using Left Ventricular (LV) and Left Atrial (LA) volumemeasurements to derive E, e′ and A (e.g., early and late diastolictransmittal flow and early/mean diastolic tissue velocity) measurements(block 436). In one embodiment, simulating the Doppler modalitymeasurements may be optional and may be invoked based on a softwaresetting indicating the presence of a handheld configuration and/orabsence of Doppler modality images for the current patient study.

Referring again to FIG. 4A, in one embodiment, once cardiac features areannotated during segmentation, the cardiac features are then measuredduring a measurement process (block 310 of FIG. 3), which in oneembodiment may comprises block 438-448. The process of measuring cardiacfeatures may begin by quantifying a plurality of measurements using theannotations. First, the 2D pipeline measures for the 2D imagesleft/right ventricle, left/right atriums, left ventricular outflow(LVOT) and pericardium (block 438). For the Doppler modality images, theDoppler modality image pipeline measures blood flow velocities (block440).

More specifically, for A2C, A3C, A4C, and A5C image views, volumetricmeasurements of chamber size are conducted on the systolic and diastolicframes of the video, and image processing techniques mimic a trainedclinician at measuring the volume using the method of disks (MOD). For2D Plax, PSAX (mid level), PSAX (AoV level), Subcostal, Suprasternal andIVC image views, linear measurements of chamber size and inter-chamberdistances are conducted on the systolic and diastolic frames of thevideo using image processing techniques to mimic the trained clinician.For M-mode image views, from the annotated segments of the movement ofthe Tricuspidvalve, a center line is extracted and smoothed, and thenthe peaks and valleys are measured in order to determine the minimum andmaximum deviations over the cardiac cycle. For PW image views, from theannotations of the blood flow, a mask is created to isolate parts of thewaveform. A sliding window is then run across the trace to identify onefull heart cycle, in combination with heart rate data from the DICOMtags, to use as a template. This template is then used to identify allother heart cycles in the image. Peak detection is then performed oneach cycle and then run through an algorithm to identify which part ofthe heart cycle each peak represents. For CW image views, from theannotations of the trace of the blood flow, curve fitting is performedon the annotation to then quantify the desired measurements. For PWTDIimage views, from the annotations of the movement of the tissue, a maskis created to isolate parts of the waveform. A sliding window is thenrun across the trace to identify one full heart cycle, in combinationwith heart rate data from the DICOM tags, to use as a template. Thistemplate is then used to identify all other heart cycles in the image.Peak detection is then performed on each cycle and then run through analgorithm to identify which part of the heart cycle each peakrepresents.

FIGS. 10A and 10B are diagrams graphically illustrating structuralmeasurements automatically generated from annotations of cardiacchambers in a 2D image, and velocity measurements automaticallygenerated from annotations of waveforms in a Doppler modality,respectively.

The measurement table below list the measurements that may be compiledby the echo workflow engine 12 according to one embodiment.

Measurement Table Measurement Name Measurement Description LAESV MOD A2CLeft Atrial End Systolic Volume in A2C calculation based on Method OfDiscs LAESVi MOD A2C Left Atrial End Systolic Volume in A2C calculationbased on Method Of Discs indexed to BSA LAL A2C Left Atrial End SysolicLength measured in A2C LVEDV MOD A2C Left Ventricular End DiastolicVolume in A2C calculation based on Method Of Discs LVEDVi MOD A2C LeftVentricular End Diastolic Volume in A2C calculation based on Method OfDiscs indexed to BSA LVEF MOD A2C Left Ventricular Ejection Fraction inA2C based on Method Of Discs LVESV MOD A2C Left Ventricular End SystolicVolume in A2C calculation based on Method Of Discs LVESVi MOD A2C LeftVentricular End Systolic Volume in A2C calculation based on Method OfDiscs indexed to BSA LV length A2C Left Ventricular Length measured inA2C LAESV MOD A4C Left Atrial End Systolic Volume in A4C calculationbased on Method Of Discs LAESVi MOD A4C Left Atrial End Systolic Volumein A4C calculation based on Method Of Discs indexed to BSA LAL A4C LeftAtrial End Systolic Length measured in A4C LAW A4C Left Atrial EndSystolic Width measurement in A4C LA area A4C Left Atrial Area measuredin A4C LAESV A-L A4C Left Atrial End Systolic Volume in A4C calculationbased on Area-Length method LVEDV MOD A4C Left Ventricular End DiastolicVolume in A4C calculation based on Method Of Discs LVEDVi MOD A4C LeftVentricular End Diastolic Volume in A4C calculation based on Method OfDiscs indexed to BSA LVEF MOD A4C Left Ventricular Ejection Fraction inA4C based on Method Of Discs LVESV MOD A4C Left Ventricular End SystolicVolume in A4C calculation based on Method Of Discs LVESVi MOD A4C LeftVentricular End Systolic Volume in A4C calculation based on Method OfDiscs indexed to BSA LV length A4C Left Ventricular Length measured inA4C LVAd A4C Left Ventricular Area measured at end diastole in A4C TAPSETricuspid Annular Plane Systolic Excursion DecT Deceleration Time ofearly diastolic MV transmitral flow E/A ratio Ratio of early and latediastolic transmitral flow MV-A Late diastolic transmitral flow MV-AdurDuration of late diastolic transmitral flow MV-E Early diastolictransmitral flow e′ lateral Early diastolic tissue velocity taken fromthe lateral region e′ mean Mean early diastolic tissue velocity (mean oflateral and septal region) e′ septal Early diastolic tissue velocitytaken from the septal region E/e′ lateral Ratio of early transmitralflow and early diastolic tissue velocity taken form the lateral regionE/e′ septal Ratio of early transmitral flow and early diastolic tissuevelocity taken form the septal region E/e′ mean Ratio of earlytransmitral flow and mean diastolic tissue velocity a′ lateral Latediastolic tissue velocity taken from the lateral region a′ septal Latediastolic tissue velocity taken from the septal region s′ lateralSystolic tissue velocity taken from the lateral region s′ septalSystolic tissue velocity taken from the septal region LAESV A-L biplaneLeft Atrial End Systolic Volume biplane calculation based on Area-Lengthmethod LAESV MOD biplane Left Atrial End Systolic Volume biplanecalculation based on Method Of Discs LAESVi MOD biplane Left Atrial EndSystolic Volume biplane calculation based on Method Of Discs indexed toBSA LAESVi A-L biplane Left Atrial End Systolic Volume biplanecalculation based on Area-Length method indexed to BSA LVCO MOD biplaneLeft Ventricular Cardiac Output biplane calculation based on Method OfDiscs LVEDV MOD biplane Left Ventricular End Diastolic Volume biplanecalculation based on Method Of Discs LVEDVi MOD biplane Left VentricularEnd Diastolic Volume biplane calculation based on Method Of Discsindexed to BSA LVEF MOD biplane Left Ventricular Ejection Fractionbiplane based on Method Of Discs LVESV MOD biplane Left Ventricular EndSystolic Volume biplane calculation based on Method Of Discs LVESVi MODbiplane Left Ventricular End Systolic Volume biplane calculation basedon Method Of Discs indexed to BSA LVSV MOD biplane Left VentricularStroke Volume biplane calculation based on Method of Disks AoV VmaxAortic Valve maximum Velocity AoV Vmean Aortic Valve mean Velocity AoVPmax Aortic Valve maximum Pressure gradient LVOT Vmax Left VentricularOutflow Tract maximum Velocity LVOT Vmean Left Ventricular Outflow Tractmean Velocity IVSd Inter Ventricular Septal thickness measured enddiastolic LV mass Left Ventricular mass LVIDd Left Ventricular internalDiameter measured at end diastole LVIDd index Left Ventricular internalDiameter measured at end diastole indexed to BSA LVIDs Left Ventricularinternal Diameter measured at end systole LVIDs index Left Ventricularinternal Diameter measured at end systole indexed to BSA LVMi LeftVentricular Mass indexed to BSA LVOT Left Ventricular Outflow Tractdiameter LVPWd Left Ventricular Posterior Wall thickness measured enddiastolic RWT Relative Wall Thickness LA area A2C Left Atrial Areameasured in A2C LAESV A-L A2C Left Atrial End Systolic Volume in A2Ccalculation based on Area-Length method LVAd A2C Left Ventricular Areameasured at end diastole in A2C LVAs A2C Left Ventricular Area measuredat end systole in A2C LVAs A4C Left Ventricular Area measured at endsystole in A4C RAL Right Atrial End Systolic Length RAW Right Atrial EndSystolic Width RAESV MOD A4C Right Atrial end systolic Volume in A4Ccalculation based on Method Of Discs RAESV A-L A4C Right Atrial endsystolic Volume in A4C calculation based on Area-Length method RAESViMOD A4C Right Atrial end systolic Volume in A4C calculation based onMethod Of Discs indexed to BSA RA area Right Atrial area RVIDd RightVentricular End Diastolic Internal Diameter RV area (d) RightVentricular Area (measured at end-diastole) RV area (s) RightVentricular Area (measured at end systole) LVOT Pmax Left VentricularOutflow Tract max pressure gradient LVOT Pmean Left Ventricular OutflowTract mean pressure gradient LVSV (Doppler) Left Ventricular StrokeVolume based on Doppler LVOT VTI Left Ventricular Outflow Tract VelocityTime Integral LVCO (Doppler) Left Ventricular Cardiac Output (based onDoppler) LVCOi (Doppler) Left Ventricular Cardiac Output (based onDoppler) indexed to Body Surface Area LVSVi (Doppler) Left VentricularStroke Volume (based on Doppler) indexed to Body Surface Area TR VmaxTricuspid Regurgitation maximum velocity CSA LVOT Crossectional Area ofthe LVOT Sinotub J Sinotubular junction diameter Sinus valsalva Sinusvalsalva diameter Asc. Ao Ascending Aorta diameter Asc. Ao indexAscending Aorta diameter index Sinus valsalva index Sinus valsalvadiameter indexed to BSA IVC max Inferior Vena Cava maximum diameter IVCmin Inferior Vena Cava minimum diameter IVC Collaps Inferior Vena Cavacollaps RVIDd mid Right Ventricular Internal Diameter at mid level(measured end diastole) RVOT prox Right Ventricular Outflow Tractproximal diameter RVOT dist Right Ventricular Outflow Tract distaldiameter RV FAC Right Ventricular Fractional Area Change TEI index RVAWTRight Ventricular Anterior Wall Thickness TrV-E Tricuspid valve E waveTrV-A Tricuspid valve A wave TrV E/A Tricuspid valve E/A ratio TrV DecTTricuspid valve deceleration time MV Vmax Mitral valve maximum velocityMV Vmean Mitral valve mean velocity MV VTI Mitral valve velocity timeintergal MV PHT Mitral valve pressure half time MVA (by PHT) Mitralvalve area (by pressure half time) RV e′ Early diastolic tissue velocitytaken from the right ventricular free wall region RV a′ Late diastolictissue velocity taken from the right ventricular free wall region RV s′Systolic tissue velocity taken from the right ventricular free wallregion RVCO Right Ventricular Cardiac Output ULS UnidimensionalLongitudinal Strain Ao-arch Aortic arch diameter Descending AoDescending Aortic diameter Ao-arch index Aortic arch diameter indexed toBSA Descending Ao index Descending Aortic diameter indexed to BSA LA GLS(reservoir) (A4C) Left Atrial strain during systole measured in A4C LAGLS (conduit) (A4C) Left Atrial strain during early diastole measured inA4C LA GLS (booster) (A4C) Left Atrial strain during pre atrialcontraction measured in A4C LA GLS (reservoir) (A2C) Left Atrial strainduring systole measured in A2C LA GLS (conduit) (A2C) Left Atrial strainduring early diastole measured in A2C LA GLS (booster) (A2C) Left Atrialstrain during pre atrial contraction measured in A2C LA GLS (reservoir)Left Atrial strain during systole LA GLS (conduit) Left Atrial strainduring early diastole LA GLS (booster) Left Atrial strain during preatrial contraction LVSr-e (A4C) Left Ventricular strain rate duringearly diastole measured in A4C LVSr-a (A4C) Left Ventricular strain rateduring late diastole measured in A4C LVSr-s (A4C) Left Ventricularstrain rate during systole measured in A4C LVSr-e (A2C) Left Ventricularstrain rate during early diastole measured in A2C LVSr-a (A2C) LeftVentricular strain rate during late diastole measured in A2C LVSr-s(A2C) Left Ventricular strain rate during systole measured in A2C LVSr-eLeft Ventricular strain rate during early diastole LVSr-a LeftVentricular strain rate during late diastole LVSr-s Left Ventricularstrain rate during systole LASr-e (A4C) Left Atrial strain rate duringearly diastole LASr-a (A4C) Left Atrial strain rate during late diastoleLASr-s (A4C) Left Atrial strain rate during systole LASr-e (A2C) LeftAtrial strain rate during early diastole LASr-a (A2C) Left Atrial strainrate during late diastole LASr-s (A2C) Left Atrial strain rate duringsystole LASr-e Left Atrial strain rate during early diastole LASr-a LeftAtrial strain rate during late diastole LASr-s Left Atrial strain rateduring systole AV-S (A4C) Atrio Ventricular strain measured in A4C AV-S(A2C) Atrio Ventricular strain measured in A2C AV-S Atrio Ventricularstrain Sr-Sav (A4C) Atrio Ventricular strain rate during systolemeasured in A4C Sr-Eav (A4C) Atrio Ventricular strain rate during earlydiastole measured in A4C Sr-Aav (A4C) Atrio Ventricular strain rateduring late diastole measured in A4C Sr-Sav (A2C) Atrio Ventricularstrain rate during systole measured in A2C Sr-Eav(A2C) Atrio Ventricularstrain rate during early diastole measured in A2C Sr-Aav (A2C) AtrioVentricular strain rate during late diastole measured in A2C Sr-SavAtrio Ventricular strain rate during systole Sr-Eav Atrio Ventricularstrain rate during early diastole Sr-Aav Atrio Ventricular strain rateduring late diastole LVVr-e (A4C) Left Ventricular volume rate duringearly diastole measured in A4C LVVr-a (A4C) Left Ventricular volume rateduring late diastole measured in A4C LVVr-s (A4C) Left Ventricularvolumerate during systole measured in A4C LVVr-e (A2C) Left Ventricularvolume rate during early diastole measured in A2C LVVr-a (A2C) LeftVentricular volume rate during late diastole measured in A2C LVVr-s(A2C) Left Ventricular volumerate during systole measured in A2C LVVr-eLeft Ventricular volume rate during early diastole LVVr-a LeftVentricular volume rate during late diastole LVVr-s Left Ventricularvolumerate during systole LAVr-e (A4C) Left Atrial volume rate duringearly diastole measured in A4C LAVr-a (A4C) Left Atrial volume rateduring late diastole measured in A4C LAVr-s (A4C) Left Atrial volumerateduring systole measured in A4C LAVr-e (A2C) Left Atrial volume rateduring early diastole measured in A2C LAVr-a (A2C) Left Atrial volumerate during late diastole measured in A2C LAVr-s (A2C) Left Atrialvolumerate during systole measured in A2C LAVr-e Left Atrial volume rateduring early diastole LAVr-a Left Atrial volume rate during latediastole LAVr-s Left Atrial volumerate during systole TLVd Total Leftheart volume end-diastolic TLVs Total Left heart volume end-systolicTLVd (A4C) Total Left heart volume end-diastolic measured in A4C TLVs(A4C) Total Left heart volume end-systolic measured in A4C TLVd (A2C)Total Left heart volume end-diastolic measured in A2C TLVs (A2C) TotalLeft heart volume end-systolic measured in A2C Ar Pulmonary vein Atrialreversal flow Ardur Pulmonary vein Atrial reversal flow duration DPulmonary vein diastolic flow velocity S Pulmonary vein systolic flowvelocity S/D ratio Ratio of Pulmonary vein systolic- and diastolic flowVel. RV GLS Right Ventricular Global Longitudinal Strain (mean) RV GLS(A4C) Right Ventricular Global Longitudinal Strain measured in A4C RVGLS (A2C) Right Ventricular Global Longitudinal Strain measured in A2CRV GLS (A3C) Right Ventricular Global Longitudinal Strain measured inA3C LA GLS Left Atrial Global Longitudinal Strain (mean) LA GLS (A4C)Left Atrial Global Longitudinal Strain measured in A4C LA GLS (A2C) LeftAtrial Global Longitudinal Strain measured in A2C LA GLS (A3C) LeftAtrial Global Longitudinal Strain measured in A3C RA GLS Right AtrialGlobal Longitudinal Strain (mean) RA GLS (A4C) Right Atrial GlobalLongitudinal Strain measured in A4C RA GLS (A2C) Right Atrial GlobalLongitudinal Strain measured in A2C RA GLS (A3C) Right Atrial GlobalLongitudinal Strain measured in A3C PV Vmax Pulmonary Valve maximumVelocity PV Vmean Pulmonary Valve mean Velocity PV Pmax Pulmonary Valvemaximum Pressure gradient PV Pmean Pulmonary Valve mean Pressuregradient PV VTI Pulmonary Valve Velocity Time Integral MV-Adur - ArdurDifference between late diastolic transmitral flow and pulmonary veinatrial reversal flow duration Mean % WT A2C Mean percentual WallThickening of 6 segments in A2C AA-% WT Percentile wall thickening ofapical anterior segment AA-WTd Wall thickness of apical anterior segmentin diastole AA-WTs Wall thickness of apical anterior segment in systoleAI-% WT Percentile wall thickening of apical inferior segment AI-WTdWall thickness of apical inferior segment in diastole AI-WTs Wallthickness of apical inferior segment in systole BA-% WT Percentile wallthickening of basal anterior segment BA-WTd Wall thickness of basalanterior segment in diastole BA-WTs Wall thickness of basal anteriorsegment in systole BI-% WT Percentile wall thickening of basal interiorsegment BI-WTd Wall thickness of basal interior segment in diastoleBI-WTs Wall thickness of basal interior segment in systole MA-% WTPercentile wall thickening of mid anterior segment MA-WTd Wall thicknessof mid anterior segment in diastole MA-WTs Wall thickness of midanterior segment in systole MI-% WT Percentile wall thickening of midinferior segment MI-WTd Wall thickness of mid inferior segment indiastole MI-WTs Wall thickness of mid inferior segment in systolePericardial effusion Pericardial effusion Mean % WT A3C Mean percentualWall Thickening of 6 segments in A3C AAS-% WT Percentile wall thickeningof apical antero-septal segment AAS-WTd Wall thickness of apicalantero-septal segment in diastole AAS-WTs Wall thickness of apicalantero-septal segment in systole AP-% WT Percentile wall thickening ofapical posterior segment AP-WTd Wall thickness of apical posteriorsegment in diastole AP-WTs Wall thickness of apical posterior segment insystole BAS-% WT Percentile wall thickening of basal antero-septalsegment BAS-WTd Wall thickness of basal antero-septal segment indiastole BAS-WTs Wall thickness of basal antero-septal segment insystole BP-% WT Percentile wall thickening of basal posterior segmentBP-WTd Wall thickness of basal posterior segment in diastole BP-WTs Wallthickness of basal posterior segment in systole MAS-% WT Percentile wallthickening of mid antero-septal segment MAS-WTd Wall thickness of midantero-septal segment in diastole MAS-WTs Wall thickness of midantero-septal segment in systole MP-% WT Percentile wall thickening ofmid posterior segment MP-WTd Wall thickness of mid posterior segment indiastole MP-WTs Wall thickness of mid posterior segment in systole Mean% WT A4C Mean percentual Wall Thickening of 6 segments in A4C AL-% WTPercentile wall thickening of apical lateral segment AL-WTd Wallthickness of apical lateral segment in diastole AL-WTs Wall thickness ofapical lateral segment in systole AS-% WT Percentile wall thickening ofapical septal segment AS-WTd Wall thickness of apical septal segment indiastole AS-WTs Wall thickness of apical septal segment in systole BL-%WT Percentile wall thickening of basal lateral segment BL-WTd Wallthickness of basal lateral segment in diastole BL-WTs Wall thickness ofbasal lateral segment in systole BS-% WT Percentile wall thickening ofbasal septal segment BS-WTd Wall thickness of basal septal segment indiastole BS-WTs Wall thickness of basal septal segment in systole ML-%WT Percentile wall thickening of mid lateral segment ML-WTd Wallthickness of mid lateral segment in diastole ML-WTs Wall thickness ofmid lateral segment in systole MS-% WT Percentile wall thickening of midseptal segment MS-WTd Wall thickness of mid septal segment in diastoleMS-WTs Wall thickness of mid septal segment in systole Global % WTGlobal percentual Wall Thickening of the Left Ventricle AoV Vmean AorticValve mean Velocity AoV Pmean Aortic Valve mean Pressure gradient AoVVTI Aortic Valve Velocity Time Integral AVA Vmax Aortic Valve Area(measured by max Vel.) AVA VTI Aortic valve Area (measured by VelocityTime Integral) AVAi Vmax Aortic Valve Area (measured by maximumVelocity) indexed to Body Surface Area AVAi VTI Aortic valve Area(measured by Velocity Time Integral) indexed to Body Surface Area ivrtIsoVolumic Relaxation Time LV GLS (A4C) Left Ventricular GlobalLongitudinal Strain measured in A4C LV GLS (A2C) Left Ventricular GlobalLongitudinal Strain measured in A2C LV GLS (A3C) Left Ventricular GlobalLongitudinal Strain measured in A3C LV GLS Left Ventricular GlobalLongitudinal Strain (mean)

Referring again to FIG. 4A, in one embodiment, the echo workflow engine12 maintains measurement confidence scores corresponding to the measuredleft/right ventricles, left/right atriums, LVOTs, pericardiums andmeasured velocities. The echo workflow engine 12 filters out echo imageshaving measurement confidence scores that fail to meet a threshold,i.e., low measurement confidence scores (block 442).

Measurement of cardiac features continues with calculating longitudinalstrain graphs using the annotations generated by the CNNs (block 444).Thereafter, a fifth CNN is optionally used to detect pericardialeffusion 446.

FIG. 11 is a diagram graphically illustrating measurements of globallongitudinal strain that were automatically generated from theannotations of cardiac chambers in 2D images.

Referring again to FIG. 4A, for all remaining non-filtered out data, theecho workflow engine 12 selects as best measurement data themeasurements associated with cardiac chambers with the largest volumes,and saves with the best measurement data, image location,classification, annotation and other measurement data associated withthe best measurements (block 447).

FIG. 12A is a diagram graphically illustrating an example set of bestmeasurement data 1200 based on largest volume cardiac chambers and thesaving of the best measurement data 1200 to a repository, such asdatabase 16 of FIG. 1.

Referring again to FIG. 4A, the echo workflow engine 12 then generatesconclusions by inputting the cardiac biomarker measurements and the bestmeasurement data 1200 to a set of rules based on internationalmeasurement guidelines to generate conclusions for medical decisionssupport (block 448). The following is an example rule set based onInternational cardiac guidelines in which the HF diagnosis is based on apoint system:

1) If any of the following measurement values are true:

-   -   septal e′<7 cm/s, or    -   lateral e′<10 cm/s, or    -   Average E/e′>=15, or    -   TR velocity>2.8 m/s and PASP>35 mmHg,    -   then add 2 points.

2) If any of the following measurement values are true:

-   -   LAVI>34 ml/m², or    -   LVMI>=149/122 g/m2 (m/w) and RWT>0.42,    -   then add 2 points.

3) If the patient is in sinus rhythm any of the following measurementvalues are true:

-   -   NT-proBNP>220 pg/ml, or    -   BNP>80 pg/ml,    -   then add 2 points.

4) If the patient has atrial fibrillation and any of the followingmeasurement values are true:

-   -   NT-proBNP>660 pg/ml, or    -   BNP>240 pg/ml,    -   then add 2 points.

5) If any of the following are true:

-   -   Average E/e′=9-14 or    -   GLS<16%,    -   then add 1 point.

6) If any of the following are true:

-   -   LAVI 29/34 ml/m², or    -   LVMI>115/95 g/m² (m/w), or    -   RWT>0.42, or    -   LV wall thickness>=12 mm    -   then add 1 point.

7) If the patient is in sinus rhythm and any of the following are true:

-   -   NT-proBNP 125-220 pg/ml, or    -   BNP 35-80 pg/ml,    -   then add 1 point.

8) If the patient has atrial fibrillation and any of the following aretrue:

-   -   NT-proBNP 365-660 pg/ml, or    -   BNP 105/240 pg/ml,    -   then add 1 point.

9) If total point score equals 2, 3 or 4, then determine that aDiastolic Stress Test or Invasive Haemodynamic Measurements arerequired. If total points equals 5 or more, then diagnosis a highprobability that the patient has HFpEF.

FIG. 12B is a diagram graphically illustrating the input of normal LV EFmeasurements into a set of rules to determine a conclusion that thepatient has normal diastolic function, diastolic dysfunction, orindeterminate.

Referring again to FIG. 4A, after the conclusions are generated, areport is generated and output (FIG. 3 block 314), which may compriseblocks 450-456. Report generation may begin by the echo workflow engine12 outputting the cardiac biomarker measurements and the bestmeasurement data 1200 to a JSON file for flexibility of export to otherapplications (block 450).

FIG. 13 is a diagram graphically illustrating the output ofclassification, annotation and measurement data to an example JSON file.

Referring again to FIG. 4A, a lightweight browser-based user interface(UI) displayed showing a report that visualizes the cardiac biomarkermeasurements and the best measurement data 1200 from the JSON file andthat is editable by a user (e.g., doctor/technician) for humanverification (block 452). As is well known, a lightweight web browser isa web browser that is optimized to reduce consumption of systemresources, particularly to minimize memory footprint, and by sacrificingsome of the features of a mainstream web browser. In one embodiment, anyedits made to the data are stored in the database 16 and displayed inthe UI.

In order to make clinically relevant suggestion to the user, the cardiacbiomarker measurements and the best measurement data 1200 areautomatically compared to current International guideline values and anyout of range values are highlighted for the user (block 454).

FIG. 14 is a diagram illustrating a portion of an example report showinghighlighting values that are outside the range of Internationalguidelines.

Referring again to FIG. 4A, the user is provided with an option ofgenerating a printable report showing an automated summary of MainFindings (i.e., a conclusion reached after examination) and underliningmeasurements of the patient's health (block 456).

FIG. 15 is a diagram illustrating a portion of an example report of MainFindings that may be printed and/or displayed by the user.

In one embodiment, the automated workflow of the echo workflow engine 12may end at block 456. However, in further aspects of the disclosedembodiments, the process may continue with advance functions, asdescribed below.

FIG. 4B is a flow diagram illustrating advanced functions of the echoworkflow engine. According to this embodiment, the echo workflow engine12, takes as inputs values of the cardiac biomarkers and specificmeasurements that were automatically derived using machine learning (seeblock 310), and analyzes the input measurements to determine diseasediagnosis/prediction/prognosis versus both disease and matched controlsand normal patient controls (block 460). In one embodiment, the echoworkflow engine 12 may use the disease predictions to perform diagnosisof any combination of: cardiac amyloidosis, hypertrophic cardiomyopathy,restrictive pericarditis, cardiotoxity, early diastolic dysfunction andDoppler free diastolic dysfunction assessment (block 462). A prognosisin the form of an automated score may then be generated to predictmortality and hospitalizations in the future (block 464).

Echocardiography is key for the diagnosis of heart failure withpreserved ejection fraction (HFpEF). However, existing guidelines aremixed in their recommendations for echocardiogram criteria and none ofthe available guidelines have been validated against gold-standardinvasive hemodynamic measurements in HFpEF.

According to one embodiment, the echo workflow engine 12 furthergenerates a diagnostic score for understanding predictions (block 466).Using machine learning, the echo workflow engine 12 validates thediagnostic score against invasively measured pulmonary capillary wedgepressure (PCWP), and determines the prognostic utility of the score in alarge HFpEF cohort.

In one embodiment, the echo workflow engine 12, takes as the inputsvalues, including the measurements that were automatically derived usingmachine learning workflow, and analyzes the input values using an HFpEFalgorithm to compute the HFpEF diagnostic score.

Recognizing that hypertensive heart disease is the most common precursorto HFpEF and has overlapping echocardiogram characteristics with HFpEF,echocardiogram features of 233 patients with HFpEF (LVEF≥50%) wascompared to 273 hypertensive controls with normal ejection fraction butno heart failure. An agnostic model was developed using penalizedlogistic regression model and Classification and Regression Tree (CART)analysis. The association of the derived echocardiogram score withinvasively measured PCWP was investigated in a separate cohort of 96patients. The association of the score with the combined clinicaloutcomes of cardiovascular mortality of HF hospitalization wasinvestigated in 653 patients with HFpEF from the Americas echocardiogramsub study of the TOPCAT trial.

According to one embodiment, left ventricular ejection fraction(LVEF<60%), peak TR velocity (>2.3 m/s), relative wall thickness(RWT>0.39 mm), interventricular septal thickness (>12.2 mm) and E wave(>1 m/s) are selected as the most parsimonious combination of variablesto identify HFpEF from hypertensive controls. A weighted score (range0-9) based on these 5 echocardiogram variables had a combined area underthe curve of 0.9 for identifying HFpEF from hypertensive controls.

FIGS. 16A and 16B are diagrams showing a graph A plotting PCWP and HFpEFscores, and a graph B plotting CV mortality or hospitalization for HF,respectively. Graph A shows that in the independent cohort, the HFpEFscore was significantly associated with PCWP in patients with HFpEF(R²=0.22, P=0.034). Graph B shows that a one-point increase wasassociated with a 12% increase in risk (hazard ratio [HR] 1.12; 95% Cl1.02-1.23, P=0.015) for the combined outcome after multivariablecorrection.

According to the disclosed embodiments, the echocardiographic score candistinguish HFpEF from hypertensive controls and is associated withobjective measurements of severity and outcomes in HFpEF.

Neural Network Training

According to a further aspect of the disclosed embodiments, the echoworkflow engine 12 incorporates a federated training platform foreffectively training the assortment of the neural networks employed bythe machine learning layer 200 (FIG. 2), as shown in FIG. 17.

FIG. 17 is a diagram illustrating a federated training platformassociated with the automated clinical workflow system. According to oneembodiment, the federated training platform 1700 comprises the echoworkflow engine 1701 executing on one or more servers 1702 in the cloud.As described above, the echo workflow engine 1701 comprises multipleneural networks (NNs) 1708, which may include some combination of GANsand CNNs, for example. The servers 1702 and echo workflow engine 1701are in network communication with remote computers 1703 a, 1703 b, 1703n (collectively computers 1703) located on premises at respectivelaboratories 1704 (e.g., lab 1, lab2 . . . , lab N). Each of thelaboratories 1704 maintains cardiac biomarker and echo (image) biomarkerfile archives referred to herein as cardiac and echo biomarker files1706 a, 1706 b . . . , 1706N (collectively echo image files 1706) ofpatient cohorts. For example, the laboratories 1704 may comprise ahospital or clinical lab environment where Echocardiography is performedas a diagnostic aid in cardiology for the morphological and functionalassessment of the heart. When performing echocardiography and takingmanual measurements, doctors typically select a small subset of theavailable videos from the echo image files, and may only measure abouttwo of the frames in those videos, which typically may have 70-100 imageframes each. In addition, it is believed cardiac biomarkers have yet tobe analyzed by machine learning.

To increase the accuracy of the neural networks comprising the echoworkflow engine, it would be desirable to make use of the each lab'scardiac and echo biomarker files 1706 as training data for machinelearning. However, some or all of the labs 1704 may treat the image filearchives as proprietary (graphically illustrated by the firewall), andthus do not allow their cardiac and echo biomarker files 1706 to leavethe premises, which means the cardiac and echo biomarker files 1706 areunavailable as a source of training data.

According to another aspect of the disclosed embodiments, the federatedtraining platform 1700 unlocks the proprietary cardiac and echobiomarker files 1706 of the separate laboratories 1704. This is done bydownloading and installing lightweight clients and a set of NNs 1708 a,1708 b, 1708 c on computers 1703 a, 1703 b, 1703 c (collectivelycomputers 1703) local to the respective labs 1704. More specifically,lightweight client executing on computer 1703 a of a first lab (Lab 1)accesses the first lab's cardiac and echo biomarker files 1706 a anduses those cardiac and echo biomarker files 1706 a to train the NNs 1708a and upload a first trained set of NNs back to the server 1702 aftertraining. The first set of trained NNs 1708 are then trained at a secondlab (e.g., lab 2) by downloading the lightweight clients and NNs 1708 bthe computer 1703 b located at the second lab 2. The lightweight clientexecuting on the computer 1703 b of the second lab can then access thesecond lab's cardiac and echo biomarker files 1706 b and use thosecardiac and echo biomarker files 1706 b to continue to continue to trainthe NNs 1708 b and to upload a second trained of NNs set back to theserver 1702. This process may continue until the NN's complete trainingat the last lab N by the lightweight client executing on the computer14N of the last lab N to access lab N's cardiac and echo biomarker files1706N to train the NNs and to upload a final trained set of neuralnetworks to the server 1702. Once uploaded to the server 1702 the finaltrain set of neural networks are then used in analysis mode toautomatically recognize and analyze the cardiac and echo biomarkers inthe patient studies of the respective labs 1704.

The federated training platform 1700 results in a highly trained set ofNNs 1708 that produce measurements and predictions with a higher degreeof accuracy. Another benefit is that federated training platform 1700unlocks and extracts value from the existing stores of cardiac and echobiomarker data. The cardiac and echo biomarker files 1706 from thelaboratories 1704 previously represented vast numbers of cardiac andecho biomarker data from past patient studies and clinical trials thatsat unused and unavailable for machine learning purposes because theimages are unstructured, views are un-labelled, and most of the imageswere ignored. Through the federated training platform 1700, these unusedand unavailable echo images are now processed by the lightweight clientof the echo workflow engine 1701 to create labelled echo images that arestored in structured image databases 1710 a, 1710 b . . . , 170N, ateach of the labs 1704, which is a necessary prerequisite for any machinelearning training or machine learning experiments performed on theimages. In one embodiment, the structured image databases remain locatedon premises at the individual labs 1704 to comply with the securityrequirements (of the labs 1704 and/or the echo workflow engine 1701).

Accordingly, the federated training platform 1700 provides access tostructured cardiac and echo biomarker databases 1710 in multiple lablocations to allow distributed neural network training and validation ofdisease prediction across multiple patient cohorts without either theoriginal cardiac biomarker data in files 1706 or the labelled echo imagefiles in the structured database 1710 ever having to leave the premiseof the labs 1704.

AI-Based Workflow Engine to Enhance Mobile Ultrasound Devices

According to yet another aspect of the disclosed embodiments, the echoworkflow engine is configured to enhance functioning of a mobileultrasound device 24 (FIG. 1). Commercially available mobile orhand-held ultrasound devices 24 are typically not available withartificial intelligence capabilities or analysis, and simply captureechocardiogram images. In this embodiment of the clinical workflowsystem, the echo workflow engine may be implemented with or withoutcardiac biomarker measurements and/or federated training.

FIG. 18 is a diagram illustrating an automated clinical workflow systemembodiment where the echo workflow engine is configured to provide amobile ultrasound device 24 with automatic recognition and measurementsof 2D and Doppler modality Echocardiographic images. In this embodiment,the echo workflow engine may run in the standalone configuration inwhich the echo workflow engine executes within a computer 1814. In theembodiment shown, the computer 1814 is implemented as a tablet computer,but the computer 1814 may comprise any computer form factor.Alternatively, the echo work engine may run in a connected configurationcomprising the echo workflow engine 1812A executing on one or moreservers 1834 in communication over network (e.g., the Internet) 1826with an echo workflow client 18128 executing on computer 1814. As usedherein, the standalone embodiment for the echo workflow engine and theconnected configuration will be referred to collectively as echoworkflow engine 8112.

In one embodiment, the computer 1814 is in electronic communication(wirelessly or wired) with peripheral devices such as an ultrasoundimaging device, or simply ultrasound device. In one embodiment, theultrasound device may comprise a mobile ultrasound device 1824, but anyform factor may be used. The mobile ultrasound device 1824 includes adisplay device, a user interface (UI), and a transducer or probe thatcaptures echocardiogram images of a patient's organ (e.g., a heart). Asthe transducer captures echocardiogram images, the images are displayedin the UI on the display device in real time. In one embodiment, themobile ultrasound device 1824 may comprise a tablet computer, a laptopor a smartphone. The computer 14 may be located in a hospital orclinical lab environment where Echocardiography is performed as adiagnostic aid in cardiology for the morphological and functionalassessment of the heart.

According to one aspect of the disclosed embodiments, the echo workflowengine 1812 enhances the mobile ultrasound device 1824 by providingsimultaneous acquisition and AI recognition of the echocardiogram (echo)images. In embodiments, software is either added to the mobileultrasound device 1824 or existing software of the mobile ultrasounddevice 1824 is modified to operate with the echo workflow engine 1812.That is the mobile ultrasound device 1824 may be modified to transmitecho images to the echo workflow engine, and receive and display theview classifications and the report, as described in FIG. 19. In yetanother embodiment, the echo workflow engine 1812 may be incorporatedinto, and executed within, the mobile ultrasound device 1824 itself.

FIG. 19 is a flow diagram for enhancing an ultrasound device to performAI recognition of echocardiogram images. The process may begin by theecho workflow engine receiving the echo images (which may includevideos) that are captured by the mobile ultrasound device 1824 anddisplayed in a user interface (UI) of the ultrasound device (block1900).

FIG. 20A is a diagram illustrating acquisition of echo images by themobile ultrasound device 1824. As described above, during a typicalpatient echocardiogram exam (referred to as a study), a user (e.g., asonographer, a technician or even an untrained user), places one or moreprobes of the ultrasound imaging device 24 against a patient's chest tocapture 2D echo images of the heart to help diagnose the particularheart ailment. Measurements of the structure and blood flows aretypically made using 2D slices of the heart and the position of themobile ultrasound device 24 is varied during an echo exam to capturedifferent anatomical sections of the heart from different viewpoints.The technician has the option of adding to these 2D echo images awaveform captured from various possible modalities including: continuouswave Doppler, m-mode, pulsed wave Doppler and pulsed wave tissueDoppler.

In the standalone configuration embodiment, the echo images may bestored as a patient study in database 16 and image file archive 18(FIG. 1) by the computer 1814, the mobile ultrasound device 1824 orboth. In the connected configuration embodiment, the echo workflowclient 1812B may store the echo images or otherwise transmit the echoimages over the network 1826 to the echo workflow engine 1812 on theservers 1843 for storage as the patient study.

Referring again to FIG. 19, once the echo images are received, the echoworkflow engine 1812 processes the echo images using one or more neuralnetworks to automatically classify by view type, i.e., which angle ofthe heart is captured (block 1902).

The echo workflow engine 1812 then provides even an untrained userreal-time feedback by simultaneously displaying the view type of theecho images in the UI of the mobile ultrasound device 1824 along withdisplay of the echo images (block 1904). See for example, viewclassification blocks 304 and 306 in FIG. 4A.

FIG. 20B is a diagram of a user interface displayed by the mobileultrasound device 1824. The UI 2000 displays an echo video (e.g., aseries of echo images) as normal, but according to one embodiment, theUI 2000 is further configured to display real-time feedback to the useralongside the echo images (shown), or overlaid on the echo images. Theecho workflow engine 1812 uses AI to automatically perform echo imageanalysis to detect and display the current view type 2002, which in theexample shown is “PLAX”. After all the required views are captures, theUI may also display an option 2004 for the user to view a report of echoimage measurements generated by the echo workflow engine 1812.

Referring again to FIG. 19, the process continues with the echo workflowengine 1812 generating a report showing calculated measurements offeatures in the echo images (block 1906). The echo workflow engine 1812then displays the report showing the calculated measurements on adisplay device (block 1908). In one embodiment, the workflow client 1812b may transmit the report showing the calculated measurements to theultrasound device 824 for display in the UI.

As described above, the calculated measurements may be generated by theneural networks segmenting regions of interest in the 2D images toproduce segmented 2D images, segmenting the Doppler modality images togenerate waveform traces to produce segmented Doppler modality images,and using both the segmented 2D images and the segmented Dopplermodality images to calculate measurements of cardiac features for bothleft and right sides of the heart.

FIGS. 20C, 20D and 20E are diagrams illustrating example pages of thereport showing calculated measurements of features in the echo images.The report may be displayed in the UI 2006 of the mobile ultrasounddevice 1824. FIG. 20C shows that the report may comprise multiple pagesof various types of information. In an embodiment, page 1 of the reportmay include an ID of the patient, processing and visitation dates, aMain findings section showing of features and conclusions (e.g., normal,abnormal and the like), and a running video of an echo image. In oneembodiment, the UI 206 may display the report in a scrollable windowthat allows the user to scroll down to view various pages of the report.

FIG. 20D show page 2 of the report, which may display featuremeasurements for the left ventricle. In one embodiment, the report maydisplay a UI control next to one or more of the features for the user toselect to display additional details. For example, in response to theuser clicking on a UI control next to “LVEF MOD biplane”, the report maydisplay information shown in FIG. 20E.

FIG. 20E shows that responsive to the user selecting “LVEF MOD biplane”,a window expands to display “ASE and EACI 2015Guidelines—Recommendations for Cardiac Chamber Quantification” inaddition to a corresponding echo image from the patient. The user mayclick on “Male” or “Female” to see different value ranges. If the userselects the feature “LVPWd”, the report may display a full screen video,rather than an expandable window.

FIG. 21 is a flow diagram showing processing by the echo workflow enginein the connected configuration comprising the echo workflow client 1812Bin network communication with the echo workflow engine 1812A executingon one or more servers 1834. The process may begin by the echo workflowclient 1812B receiving a user command to connect to the mobileultrasound device 1824 (block 2100). In one embodiment, the echoworkflow client 1812B may display a list of available devices viaBluetooth for user selection. After a wireless connection is establishedwith the mobile ultrasound device 1824, the echo workflow client 1812Breceives a command from the user to record echo images and transmits arecord command to the mobile ultrasound device 1824 (block 2102).

Responsive to receiving the record command, the mobile ultrasound device1824 initiates echo image recording and transmits captured echo imagesto the echo workflow client 1812B for screen sharing (block 2104). Theecho workflow client 1812B receives the echo images from the mobileultrasound device 1824 and transmits the echo images to the echoworkflow engine 1812A (block 2106). In one embodiment, the echo imagesmay be stored and associated with a patient study by either the echoworkflow client 1812B or the echo workflow engine 1812A.

In one embodiment, the echo workflow engine 1812A may be distributedacross a prediction server and a job server (not shown). The echoworkflow client 18128 may be configured to send single echo images tothe prediction server and send videos to the job server for viewclassification.

The echo workflow engine 1812A automatically classifies the echo imagesby view type and transmits the view types back to the echo workflowclient 1812B (block 2108). The echo workflow client 1812B receives theview types and forwards the view types to the mobile ultrasound device1824 (block 2110). The mobile ultrasound device 1824 receives anddisplays the view types in a UI along with the captured echo images(block 2112).

The echo workflow client 1812B determines if there are more view typesto be captured (block 2014). In one embodiment, the echo workflow client18128 may utilize a predetermined workflow or a list of view types tocapture. Responsive to determining there are more view types to becaptured, the echo workflow client 1812B may transmit a prompt to themobile ultrasound device 1824 for the user to capture the next view type(block 2116). The mobile ultrasound device 1824 receives and displaysthe prompt in the UI alongside the captured echo images (block 2118).

Responsive to determining there are no more view types to capture (block2114), the echo workflow client 18128, transmits a request to the echoworkflow engine 1812A for a report showing calculated measurements offeatures in the echo images (block 2120). The echo workflow engine 1812Aperforms AI workflow processing of the recorded images (block 2122), andgenerates and returns the report (block 2124). Responsive to receivingthe report, the echo workflow client 1812B or the mobile ultrasounddevice 1824 may display the report showing the calculated measurementson a display device (block 2126).

In one embodiment, the echo workflow client 1812B displays the report onthe display device of the computer 1814. For example the echo workflowclient 1812B may display a UI component, such as a “report” button orother control for user selection that causes display of the report. Inanother embodiment, the echo workflow client 1812B may, instead of or inaddition to, transmit the report to the mobile ultrasound device fordisplay in the UI of the mobile ultrasound device 1824.

Accordingly, the advantages of the present embodiment provide a mobiletool with automated, AI based decision support and analysis of both 2Dand Doppler echocardiogram images, thereby providing caregivers with aversatile, point-of care solution for triage, diagnosis, prognosis andmanagement of cardiac care.

AI-Based Guidance for an Ultrasound Device to Improve Capture of EchoImage Views

Proper transducer placement and manipulation are required to optimizeultrasound images. The placement and manipulation of the transducer willdiffer with each patient depending on the patient's physical build andthe position of the heart in the chest. A subtle change in probeposition and manipulation can significantly impact the quality of theimage and automatic recognition of the view type. As described abovewith respect to FIG. 20A, the UI 2000 of the mobile ultrasound deviceconventionally displays echo images, but is further configured todisplay along with the echo images real-time feedback to the user.

According to the present embodiment, the real-time feedback furtherincludes AI-based guidance for the ultrasound device to improve captureof echo image view types. In one embodiment, the AI-based guidance mayinclude continuously attempting AI recognition of the echo images asthey are being captured combined with displaying in the UI of the mobileultrasound device 1824 feedback indications to the user of whichdirections to move the probe 24′ so the probe 24′ can be placed in acorrect position to capture and successfully recognize the echo image.

In embodiments, software is either added to the mobile ultrasound device1824 or existing software of the mobile ultrasound device 1824 ismodified to operate with the echo workflow engine 1812. That is themobile ultrasound device 1824 may be modified to transmit echo images tothe echo workflow engine, and receive and display the view typeclassifications. In yet another embodiment the echo workflow engine 1812may be incorporated into, and executed within, the mobile ultrasounddevice 1824 itself.

FIG. 22A illustrates a flow diagram of an AI-based guidance process foran ultrasound device to improve capture of echo image views. The processmay begin by the echo workflow engine 1812 executing on a processor andreceiving the echo images that are displayed in the UI of the mobileultrasound device 1824 (block 2202). The echo workflow engine 1812processes the echo images using one or more neural networks tocontinuously attempt to automatically classify the echo images by viewtype and generates corresponding classification confidence scores (block2204).

The echo workflow engine 1812 simultaneously displays the view type ofthe echo images in the UI of the ultrasound device along with the echoimages (block 2206). In one embodiment, the confidence scores may alsobe displayed.

According to the disclose embodiments, the echo workflow engine 1812displays in the UI of the mobile ultrasound device feedback indicationsto the user, including which directions to move a probe of the mobileultrasound device so the probe can be placed in a correct position tocapture and successfully classify the echo image (block 2208).

FIG. 22B is a diagram of a user interface displayed by the mobileultrasound device 1824 and is similar to the diagram shown in FIG. 20Bwhere the UI 2130 displays a series of echo images (e.g., a video) 2132in the UI 2130 and simultaneously displays real-time feedbackindications 2134 in the UI 2130 alongside the echo images, or overlaidon the echo images 2132. According to the present embodiment, the echoworkflow engine 1812 uses AI to not only automatically perform echoimage analysis to detect and display the current view type, e.g. “PLAX”,but also to detect and display feedback indications 2134 of whichdirections to move the probe 24′ of the mobile ultrasound device 1824.

In embodiments, the feedback indications 2134 can be displayed as anycombination of alphanumeric characters and graphical objects, audio, andany combination thereof. For example, the feedback indications 2134 maycomprise text, symbols, icons, emoji's, pre-recorded digital voiceinstructions, or any combination thereof.

The feedback indications may further be displayed as both informationalguidance 2134A, directional guidance instructions 2134B, and acombination thereof as shown. Informational guidance 2134A relates todisplay of information and may include setup instructions, a list ofview types to be captured, the view type currently being captured,confidence scores of the view types, remaining view types to becaptured, all view types captured, and any combination thereof. FIG. 22Bshows an example where the informational guidance 2134A comprises thecurrent view being acquired is “PLAX, a percent completion of theacquisition, and a list of view types to be acquired.

Directional guidance instructions 2134B relate to instructions thatguide the user in moving the probe and may include i) macro-guidanceinstructions that instruct the user to adjust alignment or position ofthe probe to a known view type, and ii) micro-guidance instructions thatinstruct the user to optimize a view type of an echo image in progress,or a combination of both. Both the macro guidance instructions and themicro-guidance instructions may include probe movement instructions2134B such as: up/down or side-side, diagonal movements (up left/upright, down left/down right) rotation clockwise or counter-clockwise,tilt up/down or side-side, hold, or a combination thereof. FIG. 22Bshows an example where the directional guidance instructions 2134Bcomprises an instruction to “Hold Still, Acquiring Image”. One or moreof the directional guidance instructions 2134B may be associated with adistance value, e.g., “Move the probe to the left 1 inch”, or a timevalue, e.g., “Hold the probe still for 6 seconds”.

FIGS. 22C-22E are diagrams showing a progression of feedback indications2134 continuing from FIG. 22B. FIG. 22C shows the informational guidance2134A has been updated to include indications that the current view typebeing captured is “A4C”, the percent completion of the acquisition, andthe list of view types to be acquired, where a checkmark is displayednext to the captured view type FLAX to indicate that view type has beensuccessfully captured and recognized. The directional guidanceinstructions 2134B have been updated to display to “Please Adjust Probe”along with a probe movement instruction comprising a graphical clockwisearrow.

FIG. 22D shows the informational guidance 2134A has been updated toinclude indications that the current view type being captured is “A2C”,the percent completion of the acquisition, and the list of view types tobe acquired, where a checkmark is displayed next to the captured viewtype PLAX and A4C to indicate that these view types have beensuccessfully captured and recognized. The directional guidanceinstructions 2134B have been updated to display “Please Adjust Probe”along with a probe movement instruction comprising a graphical twistingleft arrow.

FIG. 22E shows the informational guidance 2134A has been updated toinclude indications that the all “All View Types Acquired!”, a blankpercent completion of the acquisition, and the list of view typesacquired with checkmarks. The directional guidance instructions 21348are no longer needed and not displayed since the view type captures havecompleted.

Referring again to FIG. 22A further details are shown of the process fordisplaying the feedback indications to the user in the UI of the mobileultrasound device (block 2208). This process may begin by the echoworkflow engine 1812 determining if the classification confidence scoresmeet a predetermined threshold (block 2208A). For example in oneembodiment, the predetermined threshold may be greater than 60-75% andmay be configurable. Responsive to the confidence scores meeting thepredetermined threshold, the echo workflow engine 1812 displays the viewtype and a directional guidance instruction to the user to optimize theview type until successful capture (block 2208B).

Responsive to the confidence scores not meeting the predeterminedthreshold, the echo workflow engine 1812 displays an estimated view typeand a feedback instruction in a form of directional guidanceinstructions to adjust alignment or position of the probe to a knownview type (block 2108C), and this process continues until theclassification score meets the threshold.

The echo workflow engine 1812 then determines if there is another viewtype to capture, e.g., such as defined in a workflow (block 2208D). Ifso, the process continues with processing of the echo images (block2204). Otherwise, the echo workflow engine 1812 may optionally displayinformational feedback that all views have been captured and the processends (block 2208E).

Although not shown, the process may further include prior to echo imagecapture, optionally displaying feedback instructions in a form of setupinstructions in the UI of the mobile ultrasound device 1824. This may beof aid to a novice user in terms of receiving help how to properly setupthe mobile ultrasound device 1824 and initially place the probe 24′.

Accordingly, the present embodiment provides an ultrasound device userwith immediate feedback on whether images acquired in a mobile settingare captured in the correct angle and are of sufficient quality toprovide suitable measurements and diagnosis of the patient's cardiaccondition.

A method and system for implementing a software-based automatic clinicalworkflow that diagnoses heart disease and AI recognition of both 2D andDoppler modality Echocardiographic images. The present invention hasbeen described in accordance with the embodiments shown, and there couldbe variations to the embodiments, and any variations would be within thespirit and scope of the present invention. Accordingly, manymodifications may be made by one of ordinary skill in the art withoutdeparting from the spirit and scope of the appended claims.

We claim:
 1. A computer-implemented method for artificial intelligence(AI) recognition of echocardiogram (echo) images by a mobile ultrasounddevice, the method comprising: receiving, by at least one processor, aplurality of the echo images captured by the ultrasound device, theultrasound device including a display and a user interface (UI) thatdisplays the echo images to a user, the echo images comprising 2D imagesand Doppler modality images of a heart; processing, by one or moreneural networks, the echo images to automatically classify the echoimages by view type; simultaneously displaying the view type of the echoimages in the UI of the ultrasound device along with the echo images;generating a report showing calculated measurements of features in theecho images; and displaying the report showing the calculatedmeasurements on a display device.
 2. The method of claim 1, furthercomprising generating the calculated measurements using the one or moreneural networks by: segmenting regions of interest in the 2D images toproduce segmented 2D images; segmenting the Doppler modality images togenerate waveform traces to produce segmented Doppler modality images;and using both the segmented 2D images and the segmented Dopplermodality images to calculate measurements of cardiac features for bothleft and right sides of the heart.
 3. The method of claim 1, furthercomprising implementing the software component as an echo workflowengine executing on a computer, the computer in electronic communicationwith the ultrasound device.
 4. The method of claim 1, further comprisingimplementing the software component as an echo workflow engine executingon one or more servers in communication over a network with a clientcomponent executing on a computer, the computer in electroniccommunication with the ultrasound device.
 5. The method of claim 1,further comprising adding software to the ultrasound device or modifyingexisting software in the ultrasound device to receive and display theview classifications and the report.
 6. The method of claim 1, furthercomprising recording at least a portion of the plurality ofechocardiogram images in association with a patient study.
 7. The methodof claim 1, wherein the 2D images are classified by the view type by afirst neural network, the method further comprising: training the firstneural network to classify frames of the 2D images as one of: A2C, A3C,A4C, ASC, PLAX Modified, PLAX, PSAX AoV level, PSAX Mid-level, SubcostalAo, Subcostal Hep vein, Subcostal IVC, Subcostal LAX, Subcostal SAX,Suprasternal and Other.
 8. The method of claim 1, wherein the Dopplermodality images are classified by the view type by a second neuralnetwork, the method further comprising: classifying continuous wave(CW), pulsed-wave (PW), and M-mode Doppler modality images by: if anecho image file contains a waveform modality (CW, PW, PWTDI, M-mode),inputting an echo image extracted from a Doppler modality image to a CNNtrained for CW, PW, PWTDI and M-mode view classification to furtherclassify the echo image as one of: CW (AoV), CW (TrV), CW Other, PW(LVOT), PW (MV), PW Other, PWTDI (lateral), PWTDI (septal), PWTDI(tricuspid), M-mode (TrV) and M-mode Other.
 9. The method of claim 2,wherein regions of interest in the 2D images are segmented to producesegmented 2D images, the method further comprising: determininglocations where each of cardiac chamber begins and ends and generatingoutlines of heart structures.
 10. The method of claim 2, whereinsegmenting the regions of interest in the 2D images and the Dopplermodality images further comprises: defining an imaging window for eachof the echo images, and filtering out annotations that lie outside ofthe imaging window.
 11. The method of claim 2, wherein segmenting theregions of interest in the 2D images and the Doppler modality imagesfurther comprises: using the 2D images to simulate Doppler modalitymeasurements by using Left Ventricular (LV) and Left Atrial (LA) volumemeasurements to derive E, e′ and A (early and late diastolic transmittalflow and early/mean diastolic tissue velocity) measurements.
 12. Themethod of claim 2, wherein using both the segmented 2D images and thesegmented Doppler modality images to calculate for a patient studymeasurements of cardiac features for both left and right sides of theheart, further comprises: using a 2D pipeline to measure for the 2Dimages left/right ventricle, left/right atriums, left ventricularoutflow (LVOT) and pericardium; and using a Doppler modality imagepipeline to measure for the Doppler modality images blood flowvelocities.
 13. A system, comprising: a memory storing a patient studycomprising: a plurality of echocardiogram (echo) images taken by anultrasound device of a patient heart; one or more processors coupled tothe memory; and a workflow engine executed by the one or more processorsthat is configured to: receive a plurality of echocardiogram (echo)images captured by an ultrasound device, the ultrasound device includinga display and a user interface (UI) that displays the echo images to auser, the echo images comprising 2D images and Doppler modality imagesof a heart; process the echo images to automatically classify the echoimages by view type; simultaneously display the view type of the echoimages in the UI of the ultrasound device along with the echo images;generate a report showing calculated measurements of features in theecho images; and display the report showing the calculated measurementson a display device.
 14. The system of claim 13, wherein the calculatedmeasurements are generated using one or more neural networks by:segmenting regions of interest in the 2D images to produce segmented 2Dimages; segmenting the Doppler modality images to generate waveformtraces to produce segmented Doppler modality images; and using both thesegmented 2D images and the segmented Doppler modality images tocalculate measurements of cardiac features for both left and right sidesof the heart.
 15. The system of claim 13, wherein the workflow engineexecutes on a computer in electronic communication with the ultrasounddevice.
 16. The system of claim 13, wherein the workflow engine executeson one or more servers in communication over a network with a clientcomponent executing on a computer, the computer in electroniccommunication with the ultrasound device.
 17. The system of claim 13,wherein software is added to the ultrasound device or existing softwareand the ultrasound device is modified to receive and display the viewclassifications and the report.
 18. The system of claim 13, wherein afirst neural network is trained to classify frames of the 2D images asone of: A2C, A3C, A4C, ASC, PLAX Modified, PLAX, PSAX AoV level, PSAXMid-level, Subcostal Ao, Subcostal Hep vein, Subcostal IVC, SubcostalLAX, Subcostal SAX, Suprasternal and Other.
 19. The system of claim 13,wherein a second neural network classifies continuous wave (CW),pulsed-wave (PW), and M-mode Doppler modality images by: if an echoimage file contains a waveform modality (CW, PW, PWTDI, M-mode),inputting an image extracted from a Doppler modality image to a CNNtrained for CW, PW, PWTDI and M-mode view classification to furtherclassify the image as one of: CW (AoV), CW (TrV), CW Other, PW (LVOT),PW (MV), PW Other, PWTDI (lateral), PWTDI (septal), PWTDI (tricuspid),M-mode (TrV) and M-mode Other.
 20. The system of claim 14, wherein theregions of interest in the 2D images are segmented by a third neuralnetwork to determine where each begins and ends and generating outlinesof heart structures.
 21. The system of claim 14, wherein the regions ofinterest in the 2D images are segmented by a third neural network toperform an annotation post process that spline fits outlines of cardiacstructures and adjusts locations of boundary lines closer to the regionsof interest.
 22. The system of claim 14, wherein segmenting the regionsof interest in the 2D images and the Doppler modality images includesdefining an imaging window for each of the images, and filtering outannotations that lie outside of the imaging window.
 23. The system ofclaim 14, wherein segmenting the regions of interest in the 2D imagesincludes using the 2D images to simulate Doppler modality measurementsby using Left Ventricular (LV) and Left Atrial (LA) volume measurementsto derive E, e′ and A (early and late diastolic transmittal flow andearly/mean diastolic tissue velocity) measurements.
 24. The system ofclaim 14, wherein the workflow engine uses a 2D pipeline to measure forthe 2D images left/right ventricle, left/right atriums, left ventricularoutflow (LVOT) and pericardium; and uses a Doppler modality imagepipeline to measure for the Doppler modality images blood flowvelocities.
 25. A non-transitory computer-readable medium containingprogram instructions for implementing an automated workflow, which whenexecuted by a processor configure the processor for: receiving, by theat least one processor, a plurality of echocardiogram (echo) imagescaptured by an ultrasound device, the ultrasound device including adisplay and a user interface (UI) that displays the echo images to auser, the echo images comprising 2D images and Doppler modality imagesof a heart; processing, by one or more neural networks, the echo imagesto automatically classify the echo images by view type; simultaneouslydisplaying the view type of the echo images in the UI of the ultrasounddevice along with the echo images; generating a report showingcalculated measurements of features in the echo images; and displayingthe report showing the calculated measurements on a display device.